National Academies Press: OpenBook
« Previous: Chapter 7 Building and Fire Research Laboratory
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

Chapter 8

Information Technology Laboratory

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

PANEL MEMBERS

Brian W. Kernighan, Lucent Technologies, Co-Chair

Richard E. Nance, Virginia Polytechnic Institute and State University, Co-Chair

Bishnu S. Atal, AT&T Laboratories Research

Matthew Bishop, University of California, Davis

Linda Branagan, Construct Internet Design

Jaime Carbonell, Carnegie Mellon University

Josephine Cheng, IBM Santa Teresa Laboratory

John R. Gilbert, Xerox Palo Alto Research Center

Roscoe C. Giles, Boston University

Andrew S. Grimshaw, University of Virginia

Thomas P. Kehler, Adaptivity

Jon R. Kettenring, Bellcore

John W. McCredie, University of California, Berkeley

Vijayan N. Nair, University of Michigan

Lawrence O'Gorman, Veridicom, Inc.

Thomas Parenty, Sybase, Inc.

Radia Perlman, Sun Microsystems

James L. Phillips, Boeing Shared Services Group

K.K. Ramakrishnan, AT&T Laboratories Research

Ahmed N. Tantawy, IBM Corporation

Stephanie M. White, System World, Inc.

Eddie L. Zeitler, Charles Schwab & Company

Submitted for the panel by its Co-Chairs, Brian W. Kernighan and Richard E. Nance, this assessment of the fiscal year 1999 activities of the Information Technology Laboratory is based on a site visit by the panel on February 16–17, 1999, in Gaithersburg, Md., and documents provided by the laboratory.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

LABORATORY-LEVEL REVIEW

Laboratory Mission

According to laboratory documentation, the mission of the Information Technology Laboratory (ITL) is to promote the U.S. economy and public welfare by providing technical leadership for the nation's measurement and standards infrastructure for information technology (IT).

The laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations, and other infrastructure technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scalable, and interoperable.

The ITL's role includes four components: research, measurement, standards, and service. This combination of roles is unique among the laboratories at NIST, and the variety among these activities makes ITL especially heterogeneous. Each division within the laboratory has an important role to play, but the relative emphasis on each of the four components varies greatly from division to division.

Progress on integrating the eight divisions of ITL into one unified, coherent laboratory continues, and most of the divisions seem to have moved past the transitional difficulties that the panel has commented on over the past several years. In particular, the Mathematical and Computational Sciences Division has successfully addressed many of the issues related to its role within ITL; this process has been helped significantly by the appointment of a permanent division chief. Progress has been made on defining the range of the collaborative research functions of the Statistical Engineering Division into the ITL, but this task is not yet complete and will continue to require care and attention on the part of management at all levels of NIST. Statistics plays a vital role in NIST activities, both through the traditional approach of collaborating with scientists in other NIST laboratories and through the newer pathways of contributing to studies of information technology issues within ITL and working directly with industry. It is important to manage all of these interactions carefully and to understand the value of the statisticians' contributions and the need for a transition that takes into account both work programs and the individuals involved.

Technical Merit and Appropriateness of Work

The quality of the programs in the laboratory is discussed in detail in each divisional report. With few exceptions, also detailed in the divisional reports, the programs are appropriate and well aligned with the laboratory and divisional missions.

The ITL's planning process continues to improve. This year, the panel observed several examples of projects that were undertaken with a clear definition of their intended value and NIST's likely contribution, that were carried through to a suitable degree of completion, and that were terminated or scaled back gracefully when NIST's participation was no longer vital. Instances include the Software Diagnostics and Conformance Testing Division's work on role-based access control (RBAC) and virtual reality modeling language (VRML) and the Advanced Network Technologies Division's asynchronous transfer mode (ATM) testing efforts. In several

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

cases, the experiences and methodologies from completed projects will be applicable in future activities; for example, the skills and approaches learned during work on VRML will help in the new project on extensible markup language (XML).

ITL has identified several new initiatives, and fiscal support for these new undertakings will come from reprogramming, in which as much as 20 percent of laboratory funding is shifted away from current work and into new projects. In general, the panel viewed this redistribution of support as a good method of encouraging laboratory staff to build a changing mix of activities, to develop innovative proposals, and to explore new opportunities.

The largest of these new initiatives is laboratory-wide work on pervasive computing, which is a broad area containing a wide variety of topics potentially suitable for study. Which of these many topics NIST will work on still needs to be more clearly defined, and the goals of the initiative need more focus. The panel believes that ITL should concentrate on topics where NIST possesses special expertise or has a unique role to play. In general, such criteria would imply work on early standardization activities like helping to develop guidelines, coordinating consortium activities, building reference implementations, establishing test criteria and methods, and other similar projects. Since so many companies and universities are active in the general field of pervasive computing, ITL risks being seduced by intriguing research problems that could divert focus away from more appropriate NIST activities. For example, ITL may not be uniquely qualified to develop yet another protocol in this field, but the laboratory could provide significant value to the community by developing tests that verify or demonstrate deficiencies in existing protocols.

Another proposed activity is the Knowledge Exchange initiative, which would have the broad goal of helping create a value system for intellectual property. The panel believed that this activity needed significantly more clarity and focus; the projects that would be undertaken and the relationship of the work to the ITL or NIST mission were not at all clear.

Industrial Impact

As noted in last year's report, ITL's efforts to reach out to the industrial community continue to improve, and the resulting partnerships have been productive for all parties. As the laboratory's portfolio of programs changes and as industrial priorities evolve, NIST staff will have to continually reevaluate the relationships with outside institutions in order to maintain a strong connection to the constituencies that benefit from technological and metrological advances made in the ITL.

ITL has numerous projects that directly affect U.S. industries. Perhaps the most visible is the Advanced Encryption Standard (AES), which seems to be well on track to provide a worldwide replacement standard for the Data Encryption Standard, which was first approved in 1977. There are also other security-related activities that are already important and influential or hold the prospect of being so.

ITL is involved in a variety of industry consortia and groups; these activities have been an especially effective way to positively influence industry at an early stage in the evolution of a particular technical field. Successful examples include biometrics, Java Real-time and Java Numerics, speech corpora and testing, and text recognition. For very young or particularly contentious technologies in which industry consortia have not yet crystallized, NIST has utilized its reputation as a neutral party to host workshops and instigate industry-wide discussions of

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

standards. One example of this technique in action is the Electronic Book workshop held in February of 1999. This meeting was a particularly effective and highly visible forum that brought together a wide variety of disparate groups.

In addition to its interactions with industry communities, ITL continues to have a positive impact on other laboratories within NIST, where scientists rely on ITL staff for research collaborations and technical services. The Mathematical and Computational Sciences Division and the Statistical Engineering Division have a long history of supporting work in other NIST laboratories, and they continue to perform crucial services in areas such as modeling and validation of standards activities. As ITL appropriately seeks to refocus some of the mathematical and statistical work in these divisions onto information technology issues under study in other ITL divisions, it is important not to lose sight of the value of the traditional work as well.

A final element of the impact of the ITL is its provision of vital services to the entire NIST community; these services include networking, high-performance computing, computer support for desktop and workstation machines, the telephone system, and a host of other infrastructure activities. The value of these activities may not be fully appreciated, or the level of appreciation across NIST may not be accurately communicated to the staff working in these areas. The panel stresses that this part of ITL is very important and efforts to emphasize the value of this work should continue.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Laboratory Resources

Funding sources1 for the Information Technology Laboratory (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

31.4

31.5

Competence

1.0

1.5

STRS-Supercomputing

11.9

12.0

ATP

1.8

0.9

OA/NFG/CRADA

10.6

12.8

Other Reimbursable

1.5

0.6

Agency Overhead

12.0

13.6

Total

70.2

72.9

The panel is pleased that the laboratory's dependence on other agency (OA) funding is holding at a comfortable level of under 20 percent. However, at the divisional level, panel members heard several independent comments that money was very tight and that outside funding was being actively pursued to make up for real or anticipated shortfalls. Such activities should be monitored carefully, lest OA funding begin to creep up again to higher levels.

As of January 1999, staffing for the Information Technology Laboratory included 381 full-time permanent positions, of which 314 were for technical professionals. There were also 91 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The present staff are strong and qualified, but the difficulties in hiring new personnel that were noted in last year's report remain a serious issue. Over the past year, there have been several significant losses in both professional and support staff, and the current intense market demand for information technology professionals makes finding replacements difficult.2 The joint institute with the University of Maryland was expected to attract a series of potential

1  

The NIST Measurement and Standards Laboratories funding comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations, but it is allotted by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Manufacturing Extension Partnership (MEP) funding reflects support from NIST's MEP for work done at the NIST laboratories in collaboration with or support of MEP activities. Funding to support production of Standard Reference Materials (SRMs) is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding, including that for Calibration Services, is grouped under Other Reimbursable. In addition to performing research and providing calibrations and reference data, the Information Technology Laboratory also provides services to the other NIST laboratories. This work is supported through STRS funding designated specifically for the supercomputing facilities and through NIST overhead.

2  

For graduating PhDs in this area, NIST can only offer starting salaries that are substantially below the level offered by private businesses.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

collaborators on short-term visits and to provide an infusion of new, young talent through graduate students on postdoctoral appointments. In addition, it was hoped that the cooperative arrangement would supply some alternative funding arrangements. However, the agreement between NIST and the university has still not been finalized, and the delay is having a negative effect on hiring.

Overall, morale appears to be quite high; in some divisions, the improvement over previous years is significant. The panel co-chairs met with employees who were new to NIST and separately with long-time employees. Although both groups expressed some frustrations with the space available, with the geographical separation of some divisions at NIST North, and particularly with the Department of Commerce legal department, all of the employees seemed uniformly pleased to be working at NIST. Some factors that contributed to the staff's satisfaction were the quality of equipment available for their research and the access NIST employees have to experts at other government agencies, at universities, and in industry. Staff members who had worked previously in industrial settings felt that in ITL, the communication between management and staff was much better than at most companies, and this higher level of clarity about plans and goals both enabled staff to plan ahead about their projects and gave them the freedom to choose their own approaches to tackling the key issues in their work.

The NIST director's initiative to institute management training and preparation for staff throughout NIST addresses a concern that is by no means unique to ITL: There are too few people who are ready for and interested in assuming a management role, and as a result, managerial departures can leave a significant vacuum.

DIVISIONAL REVIEWS

Mathematical and Computational Sciences Division
Mission

According to division documentation, the mission of the Mathematical and Computational Sciences Division is to provide technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to U.S. industry. The division focuses on the development and analysis of theoretical descriptions of phenomena (mathematical modeling), the design and analysis of the requisite computational methods and experiments, the transformation of these methods into efficient numerical algorithms for high-performance computers, the implementation of these methods in high-quality mathematical software, and the distribution of this software to NIST and industry partners.

The panel finds that the division programs are in excellent conformance with the division mission. Projects such as Micromagnetic Modeling and Optimal Signal Sets for Wireless Communication address applied mathematics issues directly relevant to high-technology industries important to today's U.S. economy. Other projects, such as the Digital Library of Mathematical Functions and Java Numerics, are aimed at developing important enhancements in the infrastructure for modern scientific computing. These two projects, as well as the work on Optimal Signal Sets, also directly support the ITL mission of providing technical leadership for the nation's measurement and standards infrastructure for information technology.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

A number of projects involve collaborative work with other NIST laboratories, and division staff provide vital support to activities throughout NIST. Examples of such projects include modeling activities in acoustic emissions for nondestructive testing with the Building and Fire Research Laboratory; in alloy solidification processes and high-speed metal-cutting with the Materials Science and Engineering Laboratory; and in optical reflectance of surfaces with the Building and Fire Research, Physics, and Manufacturing Engineering Laboratories. Indeed, the presence of a highly qualified and professional group of mathematicians elevates the scientific potential of the entire NIST enterprise and should continue to be viewed as a fundamental part of the scientific infrastructure at NIST.

Although the Mathematical and Computational Sciences Division has several key projects that directly support the ITL mission, the panel believes that more technical interactions between staff from this division and other personnel within ITL would strengthen the instances of supportive and collaborative work. Colloquia or project reviews in which speakers from across ITL focus their presentations for an audience of nonspecialists would facilitate such interactions.

Technical Merit and Appropriateness of Work

The work of the staff in the Mathematical and Computational Sciences Division is of high quality, and many key projects define the state of the art in computational and applied mathematics. Moreover, the engagement of prominent research mathematicians from outside NIST as part-time employees or consultants helps assure strong links to the broader research community. One example of the striking benefits of this approach can be seen in the Digital Library of Mathematical Functions project, in which nationally known experts are contributing as associate editors on topics in their area of specialty. The quality of the expertise present and the work done in the division is also evident through the recognition received by division personnel, such as receipt of the Department of Commerce Bronze Medal, the Presidential Early Career Award for Scientists and Engineers (the second such award for Mathematical and Computational Sciences Division personnel in 3 years), and the external publications, professional leadership work, and other professional activities by a number of division personnel.

As noted above, the panel finds the work of the division to be very appropriate in the context of the division, laboratory, and NIST missions. The Java Numerics project (new in 1998) is an excellent example of what NIST should be doing in the age of information technology. NIST's role as neutral arbiter is built in part on the external reputation of the staff, and therefore NIST's leadership efforts are supported by remarkably strong industry buy-in. Other divisional projects are focused on more traditional problems but are no less appropriate activities for NIST. For example, the work on computing the three-dimensional dimer constant solves a problem that has been of interest to physicists for 60 years, as this constant can be used to compute thermodynamic properties, such as the specific heat of a material, from first principles. The Digital Library of Mathematical Functions project, begun in 1997, is also an exciting and appropriate project. The goal of this work is to construct a new Web-based mathematical reference source for special functions and their applications by providing the graphs, tables of numerical values, and mathematical formulations that are needed by scientists in many disciplines. However, significant resources need to be identified for the project to proceed in a timely fashion. The panel is concerned that without such resources, the project

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

could stagnate and changes in Web technology could make it difficult to efficiently complete the project without significant re-work along the way. In the Statistical Engineering Division, a similar Web-based effort is under way to produce an updated Web version of the resource known as the Handbook of Engineering Statistics.3 As currently developing, this Web publication has many well-designed interactive features. Although the content and customers of the Digital Library and the Handbook are quite different, both projects may benefit by looking at useful architectural features and layouts in the other. Moreover, where appropriate, links should be built between the Library and the Handbook.

Although impressed by the technical content of the Digital Library effort, the panel also wishes to highlight the planning that went into this project, as it has a particularly well-defined end product and plan for completion. Overall, program planning within Mathematical and Computational Sciences Division programs should be strengthened. In general, clear directions are articulated for each project; however, goals and milestones against which progress and performance can be evaluated are not evident. The panel recognizes that budget uncertainties can make such project planning difficult, but nevertheless, stronger efforts should be made in this area.

Impact of Programs

Overall, the panel sees evidence that the work done in the Mathematical and Computational Sciences Division has had a strong positive impact on industry and on the scientific community at large. However, the variety of interactions that occurs both within and outside NIST make it difficult to determine the best ways to quantitatively measure the impact of the division's projects. The panel describes below several examples of the important work being done in the Mathematical and Computational Sciences Division and the actual and potential effects observed by the panel.

The division has for several years been a pioneer in using the World Wide Web to disseminate its work. The Guide to Available Mathematical Software (GAMS) is one of NIST's most-linked-to sites4; Matrix Market is an example of how the Web can be used to disseminate SRMs; and the Digital Library of Mathematical Functions is an ambitious project in digital networked documents. Moreover, the division's practice of making its mathematical modeling codes, such as the Object Oriented MicroMagnetic Framework and Object Oriented Finite Element Software available to the public is a good policy, although it is hard to measure the actual impact such code-sharing has on the community.

During the past year the division has increased its outreach to other ITL divisions and across NIST. An example of the promising results is the new program on Optimal Signal Sets, a collaboration with staff from the Advanced Networking Division in the ITL and from NIST's Electronics and Electrical Engineering Laboratory. The personnel involved are combining forces to solve a problem of importance to industry. The panel believes that it is essential to cast a wide net to catch such hard-to-predict winning connections; more forums in which staff from divisions

3  

Natrella, M.G., Experimental Statistics, NBS Handbook 91, U.S. Government Printing Office, Washington, D.C., 1963, reprinted October 1966 with corrections.

4  

GAMS serves roughly 12,000 users per month (where the activity of a given user may generate many separate hits to the Web site).

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

across ITL can learn of each others' skills and needs will assist in finding the key collaborative projects.

The ways in which Mathematical and Computational Sciences Division projects have an impact on industry fall into two broad categories. The first type takes advantage of direct interactions with industry. This occurs either through collaborations on the evolution of a project, as in the Java Numerics work, or through the production of tools used in industry, such as GAMS and Matrix Market, both available on the Web. The second way in which divisional personnel affect industry is through their vital collaborations in mathematical modeling and scientific computing on projects based in other NIST laboratories. Some examples include work on machining processes in the Manufacturing Engineering Laboratory and the Optoelectronic Modeling project in the Electronics and Electrical Engineering Laboratory. Contributions and impacts from such work are very important to NIST and to fulfilling NIST's mission.

NIST is holding a series of workshops, “Being the Best in the World,” and the Mathematical and Computational Sciences Division hosted one on modeling and simulation in June 1998. It is more difficult to define Best in the World in this broad area than in more specific domains of metrology like mass and temperature, but the June workshop did identify NIST's valuable role in developing and disseminating software, as well as in using modeling and simulation to support measurements and standards in other disciplines.

Resources

Funding sources for the Mathematical and Computational Sciences Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

4.3

3.1

Competence

0.1

0.2

STRS-Supercomputing

0.6

0.6

ATP

0.2

0.1

OA/NFG/CRADA

0.3

0.7

Total

5.5

4.7

The division has now achieved its goal of obtaining 20 percent of its funding from outside the division. This transition has been painful, and seeking out opportunities and preparing proposals have cost the staff a great deal of time and effort. Currently, most of the external support is due to a large number of small projects. The division expects that as external funding levels off at 20 percent, the time spent on finding and winning this support will decrease. Although this may be the case, management should monitor the progress in this direction carefully.

As of January 1999, staffing for the Mathematical and Computational Sciences Division included 30 full-time permanent positions, of which 27 were for technical professionals. There were also 11 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

This is one of the smallest divisions in the ITL, and the panel is concerned that staff attrition could drop the division below critical mass. One fiscal year 1998 departure has been a particular blow due to the resulting loss in internal leadership, technical expertise, and external leadership and visibility. Overall, the division has lost a number of people in both Gaithersburg and Boulder over the past several years, and it is important that resources be devoted to replacing key personnel. Nonetheless, the panel recognizes that moving the Compression Algorithms Group out of the Mathematical and Computational Sciences Division was a wise decision, in light of the group's small size and specialized interests.

As the division begins to hire new people, the panel recommends that the Optimization and Computational Geometry Group be enhanced to provide discrete mathematics (and discrete computational methods) expertise and support for projects in information technology. Building a coherent, useful capability within the large field of discrete mathematics will be a challenge, and the development of a viable small- to medium-sized group will require strong internal synergy within the division and a clear understanding of the expertise needed to provide support in the specific fields important to ITL. One approach that may be practical is to hire postdoctoral research associates in discrete mathematics, as the panel believes that the prospect of attracting very high quality PhDs in this field is more realistic than in other areas of information technology.

Morale continues to improve in the Mathematical and Computational Sciences Division. The efforts of NIST and ITL management to communicate directly with division staff and to reassure them of the importance of the division's continuing role both within ITL and across NIST have had a very positive impact. The appointment of a permanent and highly respected division chief has also boosted morale and can clearly be seen to be providing needed stability for the division. Although much progress has been made, the panel emphasizes that communication throughout the management chain should continue to be a high priority.

Advanced Network Technologies Division
Mission

According to division documentation, the mission of the Advanced Network Technologies Division is to provide the networking industry with the best in test and measurement technology.

This is an appropriate mission statement, which accurately reflects the NIST and laboratory missions in the context of the technologies relevant to this division's work. However, the panel was somewhat concerned that not all of the projects under way in this division were completely in conformance with this mission. Occasionally, research work seemed to be started only because a staff member was interested in a particular field rather than because the potential benefits of work in that area had been evaluated in light of NIST's special niche in measurement technology. A symptom of this problem is that sometimes the goals and objectives of projects are not clearly articulated; a contributing factor could be that the process through which projects are selected did not appear to the panel to be clearly defined.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Technical Merit and Appropriateness of Work

The Advanced Network Technologies Division currently emphasizes four areas of technology: High-Speed Networking, Internetworking Technologies, Multimedia Networking, and Wireless Communications.

The panel believes that the division can play an active leadership role, especially in response to industry requests. In a collaboration between the High-Speed Networking and Wireless Communications Groups, the division has already taken on that part in the standardization efforts on the integration of Bluetooth and Jini. Another area where industry could benefit from NIST's leadership is in studying and sorting out the standards for the “last mile” (digital subscriber lines and cable access). The division's ongoing work on Local and Multichannel Multipoint Distribution Service (LMDS and MMDS) in wireless is also relevant to these questions.

Also in High-Speed Networking, the panel observed that the goals for the proposed work on Wavelength Division Multiplexing networks appear to overlap with research already ongoing in industry, especially by vendors of equipment and by service providers. The mission of NIST is to contribute to the measurement methodology, and the division should carefully consider how the proposed project fits into this mission and who the customers would be for any results before significant NIST resources are invested in this area.

In Internetworking Technologies, the panel applauds the work on the Internetwork Protocol Security (IPsec) Web-based Interoperability Test (WIT) system and on the unfortunately named Cerberus project, a prototype and reference implementation of emerging network layer IPsec technologies. The IPsec project is a collaboration with the Computer Security Division, and the tools developed at NIST will be very helpful in encouraging use of key security technologies. Future activities planned in the IPsec area include the expansion of the test systems to address the certificate protocols of the Internet Engineering Task Force (IETF) Public Key Infrastructure (PKI) working group.

The Advanced Network Technologies Division expressed interest in becoming involved in work on Voice Over Internetwork Protocol (IP) applications, an area of considerable commercial interest. Some efforts currently under way, such as the IP Quality of Service (IP QoS) work on the Integrated Services Protocol Instrument (ISPI) and on the Distributed Internet Protocol and PERformance test system (DIPPER), could support the emerging work on Voice Over IP. The division is also considering beginning useful work on Internet performance measurements under the auspices of the Cross-Industry Working Team (XIWT). The panel encourages NIST to look into ways of making the resulting Internet measurement data widely available. This project is an excellent opportunity for the Advanced Network Technologies Division to take advantage of the considerable mathematical and statistical expertise available in other ITL divisions.

Also in the area of Internetworking Technologies, the division staff seem to be aware of but not actively involved in industry debates related to IP Multicast. The panel notes that there is room in these arguments for NIST to play a more active role. The IETF has been designing protocols in this area for close to 10 years now, but the result is a bevy of complex and unscalable proposals and a politicized atmosphere in the working group that makes reasoned technical choices impossible. If NIST could carefully and objectively evaluate the competing proposals, the industry might be able to move towards a deployable multicast standard. Reliable multicast is a related area with similar problems, and again NIST might be able to facilitate

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

progress on standards by examining competing proposals. An IETF working group for reliable multicast has recently been formed, so the time to define the scope of the issues may be now.

A key element in designing effective standards is defining the appropriate problem space, and IETF is not always successful in this necessary step. Various industries would benefit greatly if NIST were involved in this activity and could ask the important questions before and during the standardization process. For instance, in multicast, the relevant issues include the size, volatility, and geographical dispersion of the groups, the total number of speakers (particularly for secure multicast), and the number of groups operating simultaneously. In determining standards for policies for interdomain routing, key topics include the range of policies to be supported and the relative priorities of the various policies.

In the Multimedia Networking area, the panel found that the continuing work on multimedia metrics is quite valuable. There has long been a need for objective measures of quality, particularly for video. Work on Motion Pictures Expert Group testing and evaluation should be encouraged, with particular emphasis on measurement and evaluation of methods for transporting video and the techniques' impact on perceptual video quality.

The division's Wireless Communications activities were in a very preliminary stage at the time of the last assessment, and the panel applauds the strong, directed growth in this area that has occurred over the past year. The group is now comparable in size to industrial research groups in this field. Through teamwork with other NIST personnel and by exploiting NIST's traditional niche in the creation of measurement methodologies, the group has the potential to have an impact on the technical directions taken in the wireless industry in the next few years.

Overall, the panel believes that protocol development now has significant commercial impact and is an important issue for this division to consider as it plans future activities. The process for protocol standardization is loosely defined. Dominant commercial interests can play an unduly powerful role, and de facto standards developed with minimal community input or technical assessment can be promulgated. Given that having a scalable, robust communications infrastructure is in the national interest, it is critical that there be an unbiased party in a position to facilitate industry-wide standards decisions. One option is for the Advanced Network Technologies Division to follow the model set by the Computer Security Division in its work on the AES. NIST could play an activist role in quantitative and formal evaluation of alternate protocol standards proposals and aggressively provide input into the standardization process. Given quantitative data to back up any recommendations, NIST staff could take advantage of their reputation as unbiased evaluators of technologies to be a credible voice in the standardization process.

Impact of Programs

The Advanced Network Technologies Division continues to be active in a variety of industry organizations, including the IETF, the ATM Forum, the XIWT, and the International Telecommunication Union. Division personnel are well respected by the staff of these standards bodies and by the communities they serve. NIST personnel were particularly involved in development of the IPsec protocol specifications; one staff member was coauthor on five of the specifications. In addition, as of early 1999, over 100 companies have obtained the IPsec reference code, demonstrating the value of this NIST-developed tool. The IPsec WIT has also

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

proved to be useful to industry. In other divisional areas, over 800 companies have accessed NIST's ATM simulator, indicating that this tool has also had an impressive effect on industry.

A final example of the impact of this division's activities is the participation of staff members in ATM traffic management, signaling, and testing. The thoroughness of the work related to testing in the Testing Working Group of the ATM Forum is a good example of NIST's ability to play a credible role in standards activities.

Resources

Funding sources for the Advanced Network Technologies Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

4.1

4.8

Competence

0.0

0.2

ATP

0.3

0.3

OA/NFG/CRADA

1.5

1.8

Total

5.9

7.1

As of January 1999, staffing for the Advanced Network Technologies Division included 30 full-time permanent positions, of which 25 were for technical professionals. There were also 7 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers, as well as 18 faculty and guest researchers from other organizations.

The division obtains some useful external funding from the Defense Advanced Research Projects Agency (DARPA) and other government agencies, but these contracts and grants appear to be somewhat short term in nature. The uncertainty associated with this funding has had a negative impact on some promising projects (such as the IP QoS work), and the panel encourages the division to find the support necessary to take these projects to completion.

Computer Security Division
Mission

According to division documentation, the mission of the Computer Security Division is to enable organizations and individuals to use information technology with the assurance and trust that the confidentiality, integrity, reliability, and availability of information resources are protected.

The Computer Security Division's mission directly aligns with the overall ITL mission. Specifically, this division's projects and programs are focused on facilitating awareness programs, security technology research, security product metrics and testing, and specific guidance publications through collaborating with industry, acting as an honest broker in various industry/government arenas, and focusing on metrics and testing. In addition, the division

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

regularly evaluates its activities against its mission, and project goals correlate directly with the aims articulated in the mission statement.

Technical Merit and Appropriateness of Work

The panel found that the Computer Security Division programs all have high technical merit and are appropriate activities for NIST to be engaged in. Examples of projects that particularly impressed the panel include the work on AES and other cryptographic standards, the effort on cryptographic standards testing, and the Public Key Infrastructure interoperability project. These programs are technically sound and of real importance to industry. The work on Common Criteria for IT Security Evaluation has also been very productive. One result of significant value is the finalization in late 1998 of the Final Draft International Standard (FDIS) 15408, which has been adopted by the International Organization for Standardization (ISO).

The National Information Assurance Partnership (NIAP), a collaboration between NIST and the National Security Agency (NSA), has produced several accomplishments from the past year that have also contributed to the overall success of the division's strong international program. NIAP is cooperating with NIST's National Voluntary Laboratory Accreditation Program to develop evaluation and validation schemes based on the Common Criteria that can be used to accredit private-sector laboratories for the testing of information security products. Division personnel are also developing Protection Profiles and Security Targets, which are codifications of the security requirements for particular industrial sectors, such as health care. Finally, NIST's staff can take significant credit for the signing of a Common Criteria Mutual Recognition Arrangement by five countries (Germany, France, Canada, the United Kingdom, and the United States) in October of 1998.

NIST's involvement with the Administration's policies on the Clipper Chip and on Encryption Key Recovery had the potential to damage NIST's reputation in the computer security community. However, since these activities have been almost totally removed from NIST's portfolio of projects, this division is once again positioned to successfully act as a honest broker for a wide variety of standards and testing functions throughout the computer security field.

Impact of Programs

Many of the Computer Security Division's programs have a significant impact on both the private and public sectors. In particular, the AES Program has had an enormously positive impact on the division's reputation. This project has engendered international participation, and the open process is expected to result in delivery of a public algorithm of significant value to many companies. This work will be a major accomplishment that NIST can be proud of for many years to come. Similarly, the completion of the Common Criteria with specific Protection Profiles and the development of Mutual Recognition agreements among participating countries will establish the Computer Security Division as a major contributor to the U.S. information security programs and standards.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Resources

Funding sources for the Computer Security Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

6.3

5.4

ATP

0.0

0.2

OA/NFG/CRADA

3.7

3.8

Total

10.0

9.4

As of January 1999, staffing for the Computer Security Division included 48 full-time permanent positions, of which 42 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The panel is concerned by two conflicting trends: declining budgets and increasing staff size. The total divisional budget has dropped from $11.3 million in fiscal year 1997 to $10.0 million in fiscal year 1998 to an estimated $9.4 million in fiscal year 1999, whereas the number of full-time permanent positions has risen from 42 to 47 to 48 over the same time period. The quality of the staff has improved significantly with this expansion, but the division is projecting a budget deficit for this fiscal year. Currently, additional funding is being sought from other government agencies to address this shortfall. Although this approach will enable the division to maintain its current staff levels, dependence on external support can introduce other problems. Outside funding is often allocated for specific deliverables that may not conform to the division's mission or mesh with the clearly articulated goals of the current and planned projects. The necessary effort involved in winning outside support and in working on the resulting externally mandated projects will almost certainly delay some of the appropriated and important divisional projects that had been planned for fiscal year 1999.

The panel supports the division's decision to conclude its work on role-based access control. Although the project has provided a very real service to the industry, there is minimal incremental benefit that could result from further work in this field. Next the division must ask a similar question about the Common Criteria development project. Management should evaluate if continued effort in this work will result in any real increase in the security/assurance of operational systems. The staffing allocated to this project could be more effective if transferred to other projects.

The recently appointed acting division chief has initiated a new organizational structure that appears to be well suited to the work style of the division. Six program supervisor positions have been established under the two group managers, and the result is clearer definition of managerial responsibilities and closer working relationships within the teams assigned to specific projects. These new supervisors also allow the group managers to focus more on division-level management issues.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Information Access and User Interfaces Division
Mission

According to division documentation, the mission of the Information Access and User Interfaces Division is to accelerate the development of technologies that allow intuitive, efficient access, manipulation, and exchange of complex information by facilitating the creation of measurement methods and standards. These technologies include the digitization and representation of multimedia data and the use of spoken and written natural language and visual interactive modalities for search and presentation of that information. Through collaboration with industry, academia, and government, the division coordinates and provides evaluation methodologies, test suites and corpora, prototypes, workshops, and standards and guidelines to enable faster transition into the commercial marketplace.

Although this mission statement is not succinct, it reflects the heterogeneous nature of this division, in which the charter encompasses speech, language, vision, and Web technologies, including the nurture of new technologies as well as the measurement and establishment of common standards. This mission is consistent with NIST's overall mission of providing standards, measurements, and infrastructure and assuring the availability of these tools.

Technical Merit and Appropriateness of Work

The Information Access and User Interfaces Division consists of four groups: Spoken Natural Language Processing, Natural Language Processing and Information Retrieval, Visualization and Virtual Reality, and Visual Image Processing.

The Spoken Natural Language Processing Group works with the Defense Advanced Research Projects Agency, the National Security Agency, and the industrial and academic spoken language research communities on the development of standardized speech databases and metrics for evaluating state-of-the-an speech recognition systems and on the coordination of benchmark tests using these standard databases. During the past decade, these tests have served as an indispensable tool in measuring the progress of automatic speech recognition and have contributed to a steady improvement in the performance of these systems.

The primary and most externally visible activity of the Natural Language Processing and Information Retrieval Groups has been the orchestration of the Text Retrieval Conference (TREC) evaluations of information retrieval technologies. TREC provides databases and evaluation software and metrics for industrial and academic institutions' use in testing information retrieval (IR) technology. This common set of metrics is widely accepted due to NIST's reputation as a neutral third party. This group uses TREC to evaluate query-based IR, information filtering, cross-language retrieval, and other IR tasks, and the results are all reported through the highly successful TREC workshops.

Recent activities include the expansion of TREC into several new areas. One example is spoken-document retrieval, which can search transcripts of news broadcasts created using speech recognition software. Another project, evaluation of automated topic detection and tracking, provides support in the precursor technologies for data mining from free text, which is a very young, very popular direction for research and development. Through the TREC work, NIST is helping to establish standards and commonly accepted evaluation methods in this hot new field.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

Another new activity for this division is the creation of a plugboard Java-based IR architecture that permits researchers to directly compare and to cross-evaluate the effects of new IR engines and technologies in the context of a commonly available standard plugboard architecture.

The activities listed above are quite beneficial to both the scientific and industrial communities. However, the panel cautions that, in light of the long-term nature of the TREC program, it is especially important for the division to be diligent in its continuing evaluation of the impact of each element of the program. It is vital to be able to shift resources from projects with diminishing returns to new initiatives. A good example of such successful reprogramming is the cessation of the “routing” TREC task, where technology innovation had apparently plateaued, and the startup of the question-answering track just instituted for the next TREC evaluation.

The Visualization and Virtual Reality Group is involved in several activities. Some of these projects have clearly established value, but others require further focusing or additional examination. One very successful program is the WebMetrics tool suite and workshops that establish automated methods for testing Web pages for compliance either with guidelines such as Institute of Electrical and Electronics Engineers (IEEE) standards or with more general rules of Web-page usability. These tools can increase the productivity of Web facilities such as electronic commerce and other applications, and the over 1,200 downloads of the NIST software underscore the impact of this work. A project in which the actual and potential impact is much less clear is the work on VRML. The panel has several concerns related to this work. First, it is unclear whether VRML will in fact become a widespread virtual reality standard. Second, fragmentation in the industry makes it very difficult to establish common evaluation metrics in this area. The buildup of internal NIST expertise in VRML-based methods has resulted in some interesting work, such as AnthroKids, a Web-accessible way to view anthropometric data on children. Other visualization projects (such as work on virtual reality for manufacturing and on the visualization of document or information spaces) show promise, but the research needs to be accompanied by a corresponding metrology in order to be useful.

The Visual Image Processing Group covers a variety of areas, including document recognition, fingerprints, mug shots, neural networks, and optical processing. NIST work in fingerprint standards and databases is widely respected, and this high-profile program has had a positive impact on law enforcement's use of biometrics. NIST is recognized as the world leader in providing test data, classification software, and standards to the FBI and many automated fingerprint identification system (AFIS) businesses. This work is highly commendable and should continue. However, NIST's mission is to support industrial as well as governmental activities, and the panel encourages ITL to consider expanding fingerprinting efforts beyond the FBI-supported AFIS-related work and into personal biometric applications. Such a project may require internal funding, but the panel expects that the interest and involvement of the biometrics industry would be high. Benchmarking and test design by a knowledgeable and respected entity like NIST would impact not only biometrics companies but also related industries such as electronic commerce, cryptography, smartcards, and network security. In addition, multimodal biometrics (such as the combination of fingerprints with voice recognition to yield a system with two independent measurements) may be an important method for achieving stronger recognition and robustness in biometric systems. With its wide experience in mug shots, fingerprints, and voice, NIST is in a unique position to evaluate the potential of this new direction.

In the work on mug shots, as for fingerprints, NIST-developed databases and standards are known, used, and respected throughout the world. A new project on face recognition of

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

terrorists seems to be a logical and appropriate activity for NIST. If work were to be done in this area, the panel encourages the division to use standard commercial recognition systems to evaluate, measure, and recommend standards. Again, the panel believes that the work on mug shots could effectively be extended beyond law enforcement applications to NIST design of benchmarking tests for commercial systems. Such work would be of great value to the personal biometric authentication industry.

The optical signal processing work is intriguing and potentially useful. Because of the combination of fingerprint, mug shot, and optical signal processing expertise available in this division, NIST 's basic research is producing unique information about the advantages and disadvantages of using optical processing to match fingerprints or mug shots. Currently, no commercial or government entities are using optical processing in this fashion, and the audience for these results is not yet defined. If the FBI funds this research, continuation of external support will serve as a measure of the project's value. However, if NIST's goal for the effort is to jump-start commercial technology, then the panel believes that careful consideration must be given to whether the project is consuming people or resources that might be better directed toward work on nearer-term studies.

Pervasive Computing is a new initiative in the ITL, and the work is based in the Information Access and User Interfaces Division. The goal of this program is to build a testbed to explore the issues encountered when numerous small, inexpensive, disparate computers must interact for a common purpose. This is an ambitious project involving a wide range of different technologies. The panel cautions against two potential pitfalls: too much time and effort devoted to building and maintaining a complex infrastructure and the loss of generality inherent in building on specific components. NIST' s effort should be focused on the problem of interconnecting and communicating among disparate computers, protocols, and bandwidths, as standards development will be particularly valuable for this nascent technology.

Impact of Programs

The Spoken Natural Language Processing Group has developed very useful measures for providing diagnostic information on the performance of speech recognition systems. By providing these important tools to the industrial and academic research communities, the group has played a key role in building and strengthening the emerging commercial industry in automatic speech recognition.

In the Natural Language Processing and Information Retrieval Group, the TREC process has played a pivotal role in helping to scale up the IR technology and transitioning it to the commercial sector. Groups that have benefited significantly from TREC include the companies designing Web search engines. In addition, many government agencies are purchasers of commercial IR technologies, and these agencies benefit from the periodic unbiased technical evaluations that occur through TREC.

In the Visualization and Virtual Reality Group, the panel believes that to have the maximum effect, projects must include work on metrology along with any basic research. In addition, the division needs to explore new ways to effectively leverage its expertise in VRML-based methods if NIST wishes to have an impact in the increasingly important areas affected by virtual reality, including the Web and electronic commerce.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

In the Visual Image Processing Group, NIST has made an important contribution to industry through work with the Biometric Consortium and on biometrics standards efforts. The panel believes that the division has a valuable pool of expertise built up through work for law enforcement applications that could be effectively and appropriately applied to more commercial elements of biometrics. Expansion into general testing work has the potential to make a large impact on industry.

Resources

Funding sources for the Information Access and User Interfaces Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

4.6

4.3

ATP

0.4

0.1

OA/NFG/CRADA

2.3

2.9

Total

7.3

7.3

As of January 1999, staffing for the Information Access and User Interfaces Division included 40 full-time permanent positions, of which 35 were for technical professionals. There were also 12 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

High Performance Systems and Services Division
Mission

According to division documentation, the mission of the High Performance Systems and Services Division is to enable effective application of high-performance computing and communications systems in support of the U.S. information technology industry and NIST by

  • Conducting research, development, and evaluation of advanced hardware and software components, new architectures, novel application technologies, and innovative measurement and test methods for improved computing performance, scalability, functionality, interoperability, flexibility, reliability and economy;

  • Serving as a testbed for R&D in high-performance computing and information technologies such as embedded computing, displays, and data storage; gaining experience in the deployment of these technologies; and developing metrics for the representative technologies;

  • Serving as a responsive, effective, mission-critical resource spanning computational, communication, mass storage, security, archival, and scientific visualization services; and

  • Providing and managing state-of-the-art computing and networking facilities, which integrate and support an enterprise-wide heterogeneous information technology environment for NIST.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

The High Performance Systems and Services Division programs are in conformance with the division, laboratory, and NIST missions. The division's projects include an array of research and testing activities, which directly support the overall NIST and ITL missions. In addition, numerous service activities enable other NIST staff to work effectively and therefore indirectly support the execution of the overall missions. The panel is very pleased to note the past year's progress by the division in aligning its activities to the stated mission and in producing a high-quality set of programs.

Technical Merit and Appropriateness of Work

The research, standards, and service programs in this division are of high technical merit, and the goals of the work are appropriate for NIST.

The research programs, though a relatively small part of the budget, are overall at the state of the art and are well connected to the appropriate communities outside NIST. The level and quality of publications by the division staff reflect the success in this area. Highlights include the Information Storage and Interconnect Systems (ISIS) project 's work on an electronic book testbed and on the new Digital TV Application Software Environment (DASE). These important projects meet an industrial need and push the state of the art in the development and anticipation of standards for new information technologies. The panel's previous concerns about the appropriateness of this division's work on the electronic book have been resolved by the refocusing of this project on testing and standards and by the growing connections between NIST staff and the industry. Work on clusters computing continues to evolve. In what the panel sees as an important and positive step, the PC cluster has moved out of the research group to become a production service facility. The research staff are now beginning to focus on Windows NT clusters. Research results are emerging from collaborations between the Scientific Applications Support staff and other NIST scientists, as can be seen in the publication of joint research papers and presentations.

The timelines and project plans presented to the panel appear to be well thought out, valuable management tools. Significant new projects, such as DASE, have started up; ongoing projects, such as the electronic book, are growing in impact; and completed projects, such as WebSubmit and the interoperable Message Passing Interface (MPI), are being appropriately concluded.

Service functions performed in this division include responsibility for high-performance computing, telephone, and networks. The division has made great progress in working with and educating the user community at NIST about parallel computing and applications development. NIST scientists and their codes have been migrated from the Cray to the Origin 2000 machines, the IBM SP-2, and the PC clusters. The increasing use of the parallel machines and the testimony of users to the panel demonstrate that the division has successfully managed this difficult transition. Support for MPI as a distributed programming standard for clusters, the IBM SP-2, and the Origin 2000 is appropriate and useful. Finally, as mentioned above, the panel commends the transition of the PC cluster out of the research phase and into production mode, where it is managed by the service group.

The poor quality of a legacy phone system has been an issue in past assessments. This year, the panel is pleased to report that a new phone system has been deployed and is a success. In networks, the building out of the system to support additional users and traffic is proceeding

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

appropriately. Work on extending and standardizing the firewall is a key activity that deserves management attention throughout NIST. The NIST systems are vulnerable to intrusion and damage until operations are brought behind a firewall.

Impact of Programs

Admirable progress was observed on the development of explicit and effective connections between division programs and industry needs. The ISIS project hosted a very successful workshop on electronic books that brought together industry leaders and highlighted NIST 's ability to create standards in new areas. Local high school students work as programmers at the laboratory on this project, and a CNN feature on these students provided significant positive public recognition for NIST. The new DASE project started in response to industry's requests for software standards for digital television applications. Indeed, the industry has high expectations for the value of NIST 's work in this area, and the resulting request for a very fast timetable may challenge the division's resources. Other examples of impact include the division's participation in NIST's work with the Biometrics Consortium and the collaborations between High Performance Systems and Services Division staff and personnel from other ITL divisions.

The work done in migrating NIST users from the Cray to more powerful parallel and distributed computing resources has been very successful and promises to provide long-term improvements in high-performance computing for scientists throughout NIST.

Resources

Funding sources for the High Performance Systems and Services Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

2.4

2.4

Competence

0.1

0.0

STRS-Supercomputing

9.5

10.0

ATP

0.5

0.1

OA/NFG/CRADA

0.4

0.3

Other Reimbursable

0.7

0.0

Agency Overhead

5.3

6.1

Total

18.9

18.9

As of January 1999, staffing for the High Performance Systems and Services Division included 75 full-time permanent positions, of which 55 were for technical professionals. There were also 22 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

It is important that the laboratory maintain an adequate level of support for high-performance computing as a service to NIST. With the Cray being phased out in early 1999, the division must be able to explore and deploy parallel systems (either shared memory or distributed or clusters) at a scale that can support the increasing needs of the laboratories. In the research area, successful coupling of division programs to fast-moving information technology industries (as occurred in the DASE project) may require flexible increments in funding in order to work on the fast time scales required.

Distributed Computing and Information Services Division
Mission

According to division documentation, the mission of the Distributed Computing and Information Services Division is to provide the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific and engineering studies, the development of administrative applications, and the broad dissemination of information.

The division provides software for scientific computing, support for local PCs and scientific workstations, electronic information dissemination services, and administrative and management applications support. An integral component of the NIST program strategy is to leverage information technology services to support its overall mission to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards. As a primary provider of information technology support to all of NIST, the division conforms to, and helps to carry out, this important mission.

Technical Merit and Appropriateness of Work

The panel observed that the overall breadth and scope of the Distributed Computing and Information Services Division programs and plans appear to match NIST's needs in information technology support.

The division is involved in several activities designed to reduce maintenance costs and hence improve effectiveness and efficiency in the NIST computing environment. Examples of the responsibilities of this division include providing electronic mail and calendaring services, Usenet news, standard software packages, file services, and workstation and domain administration. Like utilities, these services are available 24 hours per day, 7 days per week, and they are heavily used by the NIST community. For example, in 1998 the software checkout service in Gaithersburg was accessed more than 30,000 times and the one in Boulder more than 26,000 times.

In addition, the division hosts several internal and external Web and Anonymous FTP services for NIST departments. Division staff provide a secure Web architecture and consulting and training services. The division is also charged with supporting several significant administrative applications such as the software systems used for financial, personnel, and procurement management. As part of this responsibility, the organization has completed its year 2000 compliance work and tested more than 100 NIST administrative systems. Currently staff

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

members are involved in the implementation of the property system module of the Department of Commerce Administrative Management System at NIST.

The PC Support Group provides software support, end-user training, and maintenance and upgrade services for NIST hardware. Group staff also coordinate a help-desk facility and manage several site-wide licensing agreements. To monitor the quality of work in this group, staff members developed an automated customer satisfaction survey. There have been more than 1,500 responses to this survey instrument, and users have expressed a high degree of satisfaction with the services they have received. This group also assisted more than 500 users in checking and testing their local PCs for year 2000 readiness.

New programs are planned to provide increased support for Windows NT, Linux, and PC clusters and will meet the needs of customers distributed throughout NIST. Contributions to improving the security of the computing environment include upgrading operating systems, implementing a Public Key Infrastructure for NIST staff, installing intrusion countermeasures and detection software, and installing Web access to forms with digital signatures. The plan to apply the Baldrige Quality Criteria to the work of one group in this division is a positive organizational development step.

The panel received a copy of the ITL Information Technology Services Plan for NIST.5 This document lays out an excellent overall strategy and plan for providing information technology services to the whole of NIST and describes the integration of the programs of the High Performance Systems and Services Division and those of the Distributed Computing and Information Services Division. However, both in a preliminary visit to the Distributed Computing and Information Services Division and in the full panel visit, there was little or no evidence of collaborative planning between the divisions, and no references were made to this excellent strategic plan. The panel suggests that in the future the Distributed Computing and Information Services Division should relate reports about its operational programs and plans to this coordinated strategic planning document.

Impact of Programs

As noted above, the Distributed Computing and Information Services Division provides a wide range of important support services to staff throughout all of NIST. The results of the automated customer service satisfaction survey indicate that these services are well received. Further efforts are under way to determine other methods through which users could be involved in additional service-related measurements. The active involvement of division staff in educating and advising the NIST community on hardware and software upgrades seems highly desirable. Given that NIST has stated its strategic dependence on a well-functioning, state-of-the-art information technology environment, the Distributed Computing and Information Services Division's role in making this vision a reality is of central importance. Through their support of personnel throughout NIST, staff in this division have an indirect impact on industry.

5  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Report to ITL Assessment Panel January 1999, National Institute of Standards and Technology, Gaithersburg, Md., Appendix E, pp. 57-116.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Resources

Funding sources for the Distributed Computing and Information Services Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

0.6

0.5

STRS-Supercomputing

0.9

0.8

OA/NFG/CRADA

0.1

0.2

Other Reimbursable

0.6

0.6

Agency Overhead

6.3

7.0

Total

8.5

9.1

As of January 1999, staffing for the Distributed Computing and Information Services Division included 72 full-time permanent positions, of which 65 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The division is responsible for PC hardware and software support, Web server maintenance, e-mail programs and management, large-scale administrative software, and a variety of other PC-related needs for staff throughout NIST. However, staffing levels in the Distributed Computing and Information Services Division appear to be on the low side to fulfill the division's agency-wide responsibility to provide such services to the desktops of more than 1,500 scientists, engineers, managers, and their related support staffs. Apparently, NIST has a two-tiered information technology support structure in place, where individuals in the various other operating units provide local assistance to supplement the support provided by the Distributed Computing and Information Services Division staff. The panel observes that this dual support strategy does not appear to be clearly articulated or widely understood. The appropriate balance between centrally supported services (the staff in this division) and departmentally supported services (individuals providing local help) is hard to achieve in any organization. However, it is important for NIST management to understand that reasonable support must be provided to achieve the effective use of a complex distributed computing environment. If enough support is not provided from the Distributed Computing and Information Services Division because of lack of resources, individual scientists will spend a significant amount of their time serving as their own computer support staff. Determining the appropriate long-term balance between central and distributed support should be an explicit goal of senior NIST managers.

The panel continues to recommend that computer systems be upgraded every 3 to 4 years. Appropriate funding for upgrades needs to be allocated to the operating units, as a state-of-the-art organization should not be relying on 386 and 486 machines in 1999. By increasing the uniformity of hardware and software in use on the NIST campus, management will reduce the complexity of maintenance and improve the effectiveness of staff by limiting the number and types of systems and programs in which the Distributed Computing and Information Services Division must maintain expertise.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Software Diagnostics and Conformance Testing Division
Mission

According to division documentation, the mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; to participate with industry in the development of forward-looking standards; and to lead efforts for conformance testing, even at the early development stage of standards.

The division's work on designing conformance tests is state of the art and clearly fulfills the division's mission and is consistent with the goals expressed in the laboratory and NIST missions.

Technical Merit and Appropriateness of Work

The Software Diagnostics and Conformance Testing Division has done an excellent job determining which standards merit attention. The division has emphasized newer technologies, and without exception, the programs can be expected to have widespread industry impact.

A notable improvement since last year's assessment is the decision to focus Java-related efforts specifically on producing a requirements document for “Real-Time Java” and to move away from work on virtual machine interoperability, where there is already a great deal of industry activity. Research on language requirements for real-time systems and formal modeling contributes to the unique and valuable role NIST plays as a neutral party in Java language standardization efforts.

The VRML Conformance project represented an innovative and effective way to present Internet conformance tests, semantic requirements, and standards. Now that automatic test generation tools have been completed, the project has reached an ideal stopping point. The expertise developed during the VRML work will be of significant benefit to the recently initiated XML effort. Another new project, Object-Oriented Technology for Distributed Interactive Learning, aims to define specifications that allow educational objects to be shared across the Web; the division hopes that this work will have a broad impact on education and advanced learning systems.

Because software applications are embedded in many systems that affect safety and security, work that furthers software quality and reliability is imperative. Research in the division that advances software dependability includes statistical testing, formal methods, architecture description language (ADL) standardization, and failure data analysis. Formal methods work in the division is grounded in the search for practical techniques that will scale when applied to large, complex systems. Automatic generation of specification-based tests, using formal methods and a model checker, is an example of how NIST's expertise in this area can be used to benefit industry. In ADL work, advances could result in a great increase in software quality and reliability, since formal ADLs support simulation and model checking. Because the standardization of an ADL would strengthen component-based software development by the community at large, and thus provide significant productivity gains, research in this area could have a high payoff for the software industry.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

A joint project with the Statistical Engineering Division is investigating whether statistical analysis can provide assurance that software that passes conformance tests is indeed meeting specifications. This work has won some support through a NIST Director's Competence Award, but significant results depend on the availability of sufficient funding. The momentum that was observed last year has been lost because of a reduction in the amount of staff time allotted to this project.

Collection and analysis of reference data on software failures and provision of statistical methods and tools for analysis on software systems is an important contribution toward fulfilling society's dire need for higher-quality software. Difficulty in obtaining data on software failures is a problem for NIST and for forensic research in general. The panel recommends that NIST, in conjunction with industry and government, hold workshops in order to share failure data. This type of information is essential if the software industry is to learn from its mistakes.

The panel is moderately concerned about the nondisclosure restrictions surrounding the software copymarking project. The project involves a very small expenditure, may have significant industry impact, and is being done with a company that has received an ATP grant. These factors appear to make this situation acceptable for ITL involvement. However, the panel would like assurance that nondisclosure work is generally discouraged and that this project is a carefully considered exception. Among other potential drawbacks, nondisclosure agreements may limit effective participation in broader colloquia.

Impact of Programs

Division projects are currently having a positive impact on industry in a variety of ways. A large number of companies report significant benefits from the division's Computer Graphics Metafile test suite. The RBAC model or its reference implementation has been adopted by several companies, including IBM, Sybase, and Siemens Nixdorf. The NIST document, Requirements for Real-time Extensions to the JavaPlatform,6 is being used by two industry splinter groups as the basis for their Java real-time implementations, which greatly increases the likelihood that the two implementations will be interoperable. NIST's early intervention in this process may prove to have been a vital step toward the ultimate widespread use of this technology.

As part of the VRML effort, NIST formed and chaired the Conformance Working Group within the VRML Consortium. Microsoft, Netscape, SGI, Sony and other companies all are using NIST-developed conformance tests as a metric for testing VRML browsers. The VRML test suite receives 10,000 hits per month from 1,200 unique clients. Since this number greatly exceeds the number of organizations doing VRML product development, it is likely that the site is also used for other purposes, such as three-dimensional graphics education.

6  

U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Requirements for Real-time Extensions for the Java Platform: Report from the Requirement Group for Real-time Extensions for the Java Platform, Draft NIST Special Publication (April 27, 1999), National Institute of Standards and Technology, Gaithersburg, Md., 1999.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Resources

Funding sources for the Software Diagnostics and Conformance Testing Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

4.6

4.7

Competence

0.5

0.6

ATP

0.4

0.1

OA/NFG/CRADA

2.2

2.0

Other Reimbursable

0.2

0.0

Total

7.9

7.4

As of January 1999, staffing for the Software Diagnostics and Conformance Testing Division included 39 full-time permanent positions, of which 34 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The panel was impressed with the high level of enthusiasm displayed by the staff of the division. Of course, noncompetitive salaries make it difficult to attract new computer science graduates to NIST. In the past, the division has circumvented this issue by hiring students as supplemental or part-time employees, who then are inclined to join NIST permanently after graduation.

Statistical Engineering Division
Mission

According to division documentation, the mission of the Statistical Engineering Division is to advance scientific and industrial research by applying statistical methods to the collection and analysis of data critical to NIST scientists and collaborative partners in industry.

The Statistical Engineering Division is involved in an impressively broad range of activities, involving collaborations with scientists throughout the NIST laboratories. Division staff continue to provide critical support in the Standard Reference Materials and Calibration Programs that are critical to NIST's mission.

Over the past year, the division has made a concerted effort to increase collaborations with other divisions within ITL. Such projects now constitute roughly 15 percent of the division's work, still considerably below the long-term goal set by the laboratory. Nonetheless, the number of joint projects with other ITL divisions is growing, and NIST and ITL management have recognized that it is appropriate for more than half the division's effort to remain devoted to collaboration and consulting projects outside the ITL.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Technical Merit and Appropriateness of Work

The Statistical Engineering Division aims to maintain world-class expertise in statistical methods for measurement science and technology while identifying cutting-edge statistical techniques in relevant areas and demonstrating their utility. Many division programs provide clear evidence that the division continues to be successful in meeting these goals. The following six examples particularly impressed the panel:

  • The work on Inference on a Common Mean in an Interlaboratory Study is ongoing, leading-edge methodological research in an area that is of significant importance to NIST. This project won the American Statistical Association's Youden Prize in 1998.

  • A joint project with the Computer Security Division focuses on the development of a Statistical Test Suite for the Validation of Cryptographic Random Number Generators. This 2-year project involves the development both of new statistical methods and of a suite for testing existing algorithms. It appears likely that the results from this project will be incorporated into the ANSI X9.82 standard on random number generators.

  • With personnel from the Software Diagnostics and Conformance Testing Division, Statistical Engineering Division staff are working on Software Testing Protocol Comparisons and Reliability of Conformance Tests. These projects are good examples of research aimed at comparing and evaluating state-of-the-art statistical techniques in the literature, demonstrating their utility in the context of NIST-related applications, and developing new methods as necessary to fill gaps in the literature. Unfortunately, the Statistical Engineering Division appears to have scaled down the amount of resources committed to this project, and the panel notes that this has adversely affected the progress of the project.

  • In collaboration with the Optoelectronics Division of the NIST Electronics and Electrical Engineering Laboratory, the effort on Estimation of Time-Based Distortion and Alignment of Noisy Signals represents a novel methodological contribution to NIST measurement capabilities for the characterization of photodiodes.

  • A 5-year competence grant from the NIST Director's office for work on Computational Metrology of Manufactured Parts was completed this year. This effort involved contributions from the Statistical Engineering and the Mathematical and Computational Sciences Divisions in ITL, as well as the Precision Engineering Division of the NIST Manufacturing Engineering Laboratory. This joint project has led to important advances in methodology and software for calibrating coordinate measuring machines and can be expected to have a significant impact on the manufacturing industry.

  • Finally, NIST and SEMATECH (semiconductor manufacturing technology [consortium]) staff are developing and publishing the online Handbook of Engineering Statistics. This innovative project has great potential for use in training and educating scientists and engineers in industry to learn and properly utilize modern statistical methods. The handbook appears to be such a valuable teaching tool that the panel recommends that it be integrated into the Statistical Engineering Division' s in-house training for NIST scientists.

A new project on Bayesian Metrology is too young for the panel to definitively assess its value. The division just received a 5-year competence grant to support this work, which is aimed at developing new Bayesian methods to address generic metrological problems facing NIST scientists. This project could potentially provide an improved and more sophisticated alternative

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

to the current, oversimplified procedures recommended in the ISO's 1993 Guide to the Expression of Uncertainty in Measurement.

The panel observed that the Statistical Engineering Division is still struggling to find its appropriate niche within ITL. Although a “bottom-up” approach to project selection and management has worked well for the division in the past, there needs to be more careful, systematic planning if the division is to mesh seamlessly into the ITL of the future. Since there is a long list of traditional and newer demands for statistical services, priority setting must assume an increasingly important role in order for this division to achieve the highest possible impact and make the best use of its available human and fiscal resources. The panel believes that the division should develop a strategic plan that delineates future directions and includes clear guidelines on project selection and project management.

As the Statistical Engineering Division supplements its traditional work on scientific statistical consulting with newer research focused on information technology, the panel offers a cautionary note about work in this highly popular and fast-moving area. The division would be wise to concentrate on opportunities where it has a special position of advantage or a unique set of expertise. NIST is neither a “software factory” nor a “technology manufacturer” and hence should beware of falling into the trap of undertaking research that could eventually be judged by industry to be irrelevant.

Impact of Programs

Through its extensive collaborations with other divisions and laboratories at NIST, the Statistical Engineering Division has a substantial and varied indirect impact on the scientific community at large and on industry in particular. In August 1997, a Web-based service that provides reference data sets for public use was initiated. This highly popular service receives approximately 11,000 hits per month and is likely being used by vendors of statistical software to improve the numerical accuracy of their products. Division scientists continue to publish at a reasonable pace in mainstream statistical journals as well as in journals focused specifically on applications. The innovative NIST/SEMATECH online Handbook of Engineering Statistics will have a large impact on the practice of statistics among scientists and engineers, and the panel urges the division to fully leverage this project into its educational efforts within NIST.

The Statistical Engineering Division continues to provide extensive consulting support to NIST scientists for their more routine statistical investigations. Division staff conduct workshops aimed at training NIST scientists in basic statistical methods as well as workshops and conferences on more focused research topics. The director of the ITL has suggested that one technique for satisfying the long list of routine requests for statistical services is to have NIST scientists do more of the statistical work themselves, perhaps with the help of automated software packages supplied by the Statistical Engineering Division. This concept is appealing, but it is unlikely to be practical for any situations other than the most narrowly focused and repetitive tasks. However, the panel believes that targeted educational efforts, which could possibly include some automation or tailoring of software, may be a productive first step towards reducing the burden placed on the division by routine statistical requests. For example, the NIST/SEMATECH online Handbook of Engineering Statistics could be a valuable tool in such an educational effort. Regardless of the implementation details, the panel encourages NIST to continue to look into solutions for this problem.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Resources

Funding sources for the Statistical Engineering Division (in millions of dollars) are as follows:

 

Fiscal Year 1998

Fiscal Year 1999 (estimated)

NIST-STRS, excluding Competence

2.8

2.8

Competence

0.3

0.5

STRS-Supercomputing

0.5

0.0

OA/NFG/CRADA

0.0

0.9

Total

3.6

4.2

As of January 1999, staffing for the Statistical Engineering Division included 23 full-time permanent positions, of which 21 were for technical professionals. There were also eight nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.

The division staff includes five people who have joint appointments at NIST and as faculty at various universities. These personnel are making a significant contribution to the work of the division, and the panel strongly urges continued funding for this sort of arrangement.

To increase its impact in the information technology arena, the Statistical Engineering Division will need to consider hiring staff with backgrounds in both computer science and statistics. The current personnel have limited capabilities of this kind, as is the case in virtually all academic and industrial statistics units. Furthermore, the demand for the small pool of people with this sort of expertise is extraordinary. Special effort and considerable salary flexibility will be required to hire staff in this area. However, if increased attention is paid to priority setting and the amount of more routine work is reduced, the division will be in a position to increase the focus on projects with novel statistical aspects, which, in turn, will improve morale and make the division more attractive to inventive recruits.

One area of concern to the panel is the relative isolation of the Statistical Engineering Division in its current location at NIST North. Given the interdisciplinary nature of much of the division 's work and its extensive collaborations with other scientists at NIST, the panel strongly urges NIST management to consider relocating this division to the main campus.

MAJOR OBSERVATIONS

The panel presents the following major observations.

  • The Information Technology Laboratory (ITL) and NIST management should explicitly recognize and support the varied roles of the divisions within the ITL. Performance measurement and resource allocation must take into account the differing work profiles within the divisions. The NIST director's visits to the staff to clarify the roles of individual divisions have been very helpful and should be continued as much as possible for as long as is necessary. In particular, the panel noted that the collaborative and research functions of

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×

the Statistical Engineering Division still need to be more fully integrated into the ITL's activities and mission.

  • NIST's historic culture of openness creates a potentially serious problem when applied to its computer networks. The current firewall situation is inadequate and leaves NIST open to a potentially damaging and embarrassing intrusion. The process of ensuring that the entire organization is protected by a centrally maintained firewall system is proceeding much too slowly; strong managerial action is required to remedy this dangerous situation in an efficient and timely manner.

  • The ITL continues to refine the process by which older programs are selected for termination and new areas are chosen for funding. The panel applauds the progress that has been made so far and supports efforts to continue to improve the process.

  • Several divisions in ITL have experienced difficulties in interacting with the Department of Commerce's legal services divisions. The two examples in most desperate need of being resolved soon are the year-long failure to finalize the arrangements with the University of Maryland for a joint institute for information technology and the 2-year delay in approval for NIST to join the World Wide Web Consortium. Other cases of inefficiency, such as months taken to obtain a nondisclosure agreement, have also been observed.

  • The panel again recommends that the ITL more actively gather data on the results and consequences of its technical activities. More organized, quantitative tracking of interactions and outcomes would make it easier for NIST to justify the value of the work performed in this laboratory.

  • Information technology is a dynamic area with an extremely high demand for skilled people in all roles. NIST must find creative ways to attract and retain employees at all education and skill levels; this effort should include exploring ways to arrange increased compensation for such employees.

  • ITL management has been especially responsive to the panel's previous suggestions and to its requests for information. The panel is grateful for this excellent cooperation.

Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 187
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 188
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 189
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 190
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 191
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 192
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 193
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 194
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 195
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 196
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 197
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 198
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 199
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 200
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 201
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 202
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 203
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 204
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 205
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 206
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 207
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 208
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 209
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 210
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 211
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 212
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 213
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 214
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 215
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 216
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 217
Suggested Citation:"Chapter 8 Information Technology Laboratory." National Research Council. 1999. An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999. Washington, DC: The National Academies Press. doi: 10.17226/9685.
×
Page 218
Next: Appendixes »
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!