Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 179
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Chapter 8 Information Technology Laboratory
OCR for page 180
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 PANEL MEMBERS Brian W. Kernighan, Lucent Technologies, Co-Chair Richard E. Nance, Virginia Polytechnic Institute and State University, Co-Chair Bishnu S. Atal, AT&T Laboratories Research Mary Ellen Bock, Purdue University Allen L. Brown, Jr., Cinebase Software, Inc. Josephine Cheng, IBM Santa Teresa Laboratory Dorothy E. Denning, Georgetown University James L. Flanagan, Rutgers University Roscoe C. Giles, Boston University Andrew S. Grimshaw, University of Virginia Carl M. Harris, George Mason University Thomas P. Kehler, Conclusive Software, Inc. Sandra M. Lambert, Lambert & Associates Radia Perlman, Sun Microsystems James L. Phillips, Boeing Shared Services Group Jerome Sacks, National Institute of Statistical Sciences Ahmed N. Tantawy, IBM Corporation Raymond T. Yeh, International Software Systems, Inc. Eddie L. Zeitler, Charles Schwab & Company Submitted for the panel by its Co-Chairs, Brian W. Kernighan and Richard E. Nance, this assessment of the fiscal year 1998 activities of the Information Technology Laboratory is based on a site visit by the panel on March 17–19, 1998, and documents provided by the laboratory.
OCR for page 181
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 LABORATORY-LEVEL REVIEW Laboratory Mission The mission of the Information Technology Laboratory (ITL) is to “promote the U.S. economy and public welfare by providing technical leadership for the nation's measurement and standards infrastructure for information technology.” The Information Technology Laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations, and other infrastructural technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scalable, and interoperable. The ITL's role includes four components: research, measurement, standards, and service to both industry and other NIST laboratories. This combination of roles is unique among the laboratories at NIST. It is important that ITL management and NIST senior management recognize that ITL is an especially heterogeneous laboratory. Each division has an important role to play, but the weighting of the four components varies greatly across them. The aggregated contributions of ITL must match the needs of NIST and industry, but no uniform profile will match the individual divisions. Forcing each division into a common model would be inappropriate; instead, the individual contributions should be acknowledged and appreciated and explained to the community at large in formal mission statements, annual assessments, and day-to-day behavior. Progress has been made in integrating the collaborative research functions of the Statistical Engineering Division (SED) and Mathematical and Computational Sciences Division (MCSD) into the rest of the laboratory. The task is not yet complete, however, and there are still strains, uncomfortable situations, and morale issues. This situation requires continuing work on the part of ITL management to state clearly and consistently the importance of the divisions and what their roles are; on the part of the management and staff of the divisions to be sensitive to the importance of working within the ITL environment; and on the part of NIST management itself to recognize the NIST-wide role and importance of these divisions as a fundamental part of NIST 's mission. It is clear that ITL personnel involved in service activities on behalf of other parts of NIST (for example, providing networking or support of supercomputers) do not feel that their work is adequately recognized or appreciated outside of ITL. Because one of the attractions of support work at NIST is the chance to work with researchers on technically exciting projects, it is important that everyone involved feel that their contributions are important and appreciated. To the extent that this is not true, it will be even harder to attract good support personnel in this area. Morale and work satisfaction are two quite important factors in motivating these support staff, whose talents can command much higher salaries in the industrial and commercial sector. Technical Merit and Appropriateness of Work The quality of the programs in the laboratory is discussed in detail in each divisional report. With a few exceptions, also detailed in the divisional reports, the programs are well aligned with laboratory and divisional missions.
OCR for page 182
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 The ITL's planning process continues to improve. The laboratory has a coherent picture of its emerging goals, focused on the information technology industry in the United States and consistent with the overall NIST mission. The effects of significant and conscious planning by the laboratory are reflected in the new focus and in better-defined rules for project selection and phasing out. The laboratory has assumed an increased emphasis on both industrial needs and the unique resources and skills of the ITL. Laboratory managers and staff have made many hard decisions, and the resulting set of programs reflects a clearer vision of the future of information technology and of the role of the laboratory. The time lines that accompany this year's reports from ITL are valuable, and their continued use is encouraged. The inclusion of staffing profiles would further enhance their usefulness. Most of the work of the laboratory appears to be suitable and appropriate for its mission, though there are a few projects that should be examined more closely. In general, the criteria for selecting projects should include explicit support from industry groups or consortia relevant to ITL's mission, the need for the unique NIST role of neutral arbiter, and the need for a competence unique to ITL. The panel endorses the ITL strategy of targeting fairly young technologies, so that changes can be made before industry practices have hardened. It is in these early stages that the tests, advice on standardization, and reference implementations provided by the laboratory are most valuable to industry. The ITL's major contributions to standards will come through such early involvement, which helps voluntary standards groups define and strengthen rapidly evolving standards. The laboratory should place more emphasis on participation in such industrial standards groups and less on NIST's position as an arbiter of conformance to mature standards. Impact of Programs The ITL is unique among NIST laboratories in that services and consulting for other NIST laboratories are a major part of its program. The ITL thus has two sets of customers to address: internal NIST customers and external customers. The panel met with an ITL-selected sampling of internal customers of its support and consulting services. Overall, this group was very positive about the quality of services received and collaborations undertaken with ITL. This is in contrast to the feelings of underappreciation expressed by members of the ITL support and consulting divisions. The ITL continues to be responsive to the need for increased involvement with industry. Many divisions are focused on customers and make a real effort to seek new partners. Although there are still areas that might be improved, the focus, alertness, and conscientiousness of the effort are laudable. The laboratory has also done a good job in disseminating its results to the industrial community; innovative use of Web technology continues to be used to good effect here, both for dissemination of information and for tracking of usage patterns. The laboratory is encouraged to continue and expand such activities. The components from which the ITL was formed have a long history of industrial impact. Some of this is direct, as in the security work of the Computer Security Division and the support for test corpora (suites of tests and test data for verifying software performance) and technology evaluation conferences of the Information Access and User Interfaces Division. In these cases,
OCR for page 183
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 ITL's perceived neutrality gives it a unique opportunity to contribute. Other contributions are more indirect, as in the support provided by the Mathematical and Computational Sciences Division and the Statistical Engineering Division to research in other parts of NIST. It is important that this support be recognized when the impact of ITL is evaluated. Efforts to make the value of this support more visible to other laboratories of NIST and to their industrial contacts need to continue. Since the previous assessment, ITL has made significant progress in its tracking of industrial contacts and impact. This is a difficult but worthwhile task, as such information can be used to help focus efforts in areas that appear most important to industry and to improve the laboratory's ability to demonstrate the value of its programs to NIST management, Congress, and potential industrial customers. The use of explicit time lines for projects is also a useful addition to project tracking and planning. Laboratory Resources Funding sources1 for the Information Technology Laboratory (in millions of dollars) are presented below: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 31.2 30.4 Competence 0.7 1.0 STRS-Supercomputing 12.3 12.3 ATP 2.1 1.5 MEP 0.1 0.0 OA/NFG/CRADA 9.9 11.9 Other Reimbursable 1.9 1.6 Agency Overhead 11.0 12.5 Total 69.2 71.2 1 The NIST Measurement and Standards Laboratories funding comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations, but it is allotted by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Manufacturing Extension Partnership (MEP) funding reflects support from NIST's MEP for work done at the NIST laboratories in collaboration with or support of MEP activities. Funding to support production of Standard Reference Materials is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding including that for Calibration Services is grouped under Other Reimbursable. In addition to performing research and providing calibrations and reference data, the Information Technology Laboratory also provides services to the other NIST laboratories. This work is supported through STRS funding designated specifically for the supercomputing facilities and through NIST overhead.
OCR for page 184
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Staffing for the Information Technology Laboratory currently includes 362 full-time permanent positions, of which 295 are for technical professionals. There are also 87 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. A large majority of ITL funding is from NIST internal appropriations. This comes from three sources: the Scientific and Technical Research and Services, which is the direct appropriation from Congress for the NIST laboratories; payments from NIST-wide overhead to support services provided by the ITL to all of NIST; and Industrial Technology Services, which includes the Advanced Technology Program and the Manufacturing Extension Partnership. The remainder of the laboratory 's funding is from other agencies and from sales and services related to reference materials produced by the laboratory. Note that in the charts in this report, the portion of the STRS funding that is specifically allotted for the supercomputing facilities is listed separately. The panel was pleased that the laboratory's dependence on OA funding is running at what appears to be a comfortable level, under 20 percent. The ITL director has directed that the Mathematical and Computational Sciences Division and the SED move toward a new funding paradigm wherein 20 percent of each unit's funding comes from sources external to MCSD, e.g., from other ITL divisions or other NIST laboratories or from other government sources. If this were to develop, it would have some advantages for these divisions. It would provide a vehicle for division personnel to become familiar with, and involved with, a much broader collection of NIST projects. It would add support to develop a larger group with more technical diversity and would be in a better position to contribute to the overall mission of NIST. However, it is clear that there are formidable obstacles to making this transition, from both within and outside these two divisions. From within, many people were attracted to these units, in part, because staff have not traditionally been expected to seek funding for their own activities. To make a successful transition away from this practice, staff involved must understand the advantages of it and need for it. Moreover, groundwork must be laid at the other NIST laboratories. They traditionally have not paid for support help from mathematicians, and there are indications that they would find the transition to such support problematic due to their own budget constraints. The present ITL staff are strong and qualified, but the difficulties in hiring new personnel that were noted last year appear to be even more serious this year. Currently an intense market demand exists for information technology professionals. NIST management needs to consider whether the starting salary levels can be adjusted to be competitive in this environment. As noted above, one way to compensate for lower salaries is freedom from issues like getting grants. The Advanced Measurement Center for Information Technology (AMCIT), a joint research and development institute with the University of Maryland, is another innovation that might help bring a flow of contributors and potential future hires, such as interns, graduate students, and visiting faculty. All ITL divisions should be involved with AMCIT because its highly visible activities should provide benefits through the attraction of eminent scientists that can in turn serve to draw younger talent. The potential here is extremely important to realizing the broader mission of NIST and helping ITL to support that mission. Significant hardware investments will be needed to bring NIST up to the level of a world-class operation. The supercomputers are badly out of date, and individual computer services need continual upgrading on a short (2- to 3-year) cycle. A perception widely shared by ITL staff is that internal NIST systems for procurement, personnel, and so on, are far too slow and cumbersome, especially when compared with normal
OCR for page 185
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 business practice. Activities such as finalizing a job offer, which might take a week in industry, can take months at NIST. This puts NIST at a disadvantage in recruiting. DIVISIONAL REVIEWS Mathematical and Computational Sciences Division Mission The mission of the Mathematical and Computational Sciences Division is to provide analytical and computational methods for solving scientific, engineering, and information technology problems of critical importance to U.S. industry. This mission is discharged through a program of advanced research in selected areas of applied and computational mathematics and collaboration with technical experts in other NIST divisions, industry, and academia. In general, since the previous year, there is improved alignment of the work of MCSD with the ITL mission and its focus on metrology. MCSD has strengthened its work in this area by initiating the Digital Library of Mathematical Functions project. This is a significant undertaking and very consistent with the notion of advancing metrology and standards in the computational mathematics portion of the ITL mission. It also adds to the work in this same area represented by the Statistical Reference Datasets (StRDs) project being conducted with the SED and the MCSD's ongoing work on the Matrix Market and sparse BLAS (Basic Liner Algebra Subprograms). In each of these cases, MCSD is directly enhancing standards and metrology for the computational sciences. The larger scientific community has benefited by NIST work in this area for many years. A continued effort ought to be made to develop more projects that align with the ITL focus on metrology. The panel emphasized again the observation made a year ago that the addition of one or more discrete mathematicians would aid in this endeavor. It is also important to preserve the role of optimization in the division in light of recent staff departures in that area. The new agreement with the University of Maryland on metrology in information technology ought to be viewed—by both ITL management and MCSD personnel—as a potential vehicle to spur further such MCSD work. The panel finds the MCSD program to be in general conformance with the overall NIST mission. However, the panel also applauds the efforts of the ITL director to help link MCSD personnel with personnel from other NIST laboratories when those laboratories engage in strategic discussions with outside customers on projects involving MCSD. If others in NIST view MCSD personnel as strategic partners, it will help raise the visibility of MCSD across NIST as well as outside NIST. Technical Merit and Appropriateness of Work The work of the MCSD staff continues to be of high quality and to be highly respected in the applied mathematics community at large. The division has an unusual (but quite solid) portfolio of projects, some of which are important and innovative. As noted above, they have
OCR for page 186
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 added a new project on the digital replacement of the classical Handbook of Mathematical Functions,2 wherein the formulas, graphs, and tables are to be expanded into downloadable formats, dynamic graphics, tables on demand, reference algorithms, and so on. This work is very much in the spirit of the traditional role of the division, and its highlighting as a major new initiative of ITL is welcomed. The major continuing projects are solid as well, and the general reaction the work has received both inside and outside of NIST has been positive. Special examples of important continuing work include the Matrix Market (a Web-based database of test problems for numerical linear algebra), Micromagnetic Modeling (a collaboration with the NIST Materials Science and Engineering Laboratory to develop computational tools for micromagnetic modeling of materials), and JazzNet (the development of a “personal supercomputer” from a dedicated cluster of PCs). Impact of Programs The effectiveness of the dissemination of the division's results varies widely. The Mathematical Software Group facilitates access to mathematical software tools for NIST users and the external community via the Guide to Available Mathematical Software and other projects. The division's statistics show that the GAMS Web site continues to record a significant number of hits, a clear indication of widespread interest. Many other projects have good descriptive material available on the Web as well. However, a conscious effort focused on technology transfer would undoubtedly lead to broader usage of MCSD results within NIST. For example, MCSD's seminar series aimed at researchers across NIST might be pursued more aggressively with the objective of making other NIST scientists and engineers aware of MCSD work and how they might benefit from it. Such outreach takes time away from other ongoing work, and its benefits might take time to accrue. But over time, it would likely lead to stronger support of the division by other laboratories and ITL divisions and to a more varied set of collaborative projects as well. The division's direct impact on industry has also been most significant through the output of the Mathematical Software Group: GAMS, the Matrix Market, the sparse BLAS, and the work on the open/graphics language (Open/GL) graphics system. MCSD members have also been active professionally, with papers in refereed journals, talks at professional meetings, and professional journal editorships. Such activities are extremely important, not only as a vehicle for dissemination of results, but also as a validation of the technical quality of the work carried out and as a means for helping personnel keep abreast of research being conducted elsewhere. The division also has an impact on industry through the collaborative work done with scientists in other NIST laboratories. The efforts of the ITL director to increase the direct interaction of MCSD collaborators with industrial customers of the other NIST laboratories will, if successful, facilitate a more direct understanding of the impact of the MCSD work. Consideration should be given to finding other ways to measure such impacts. 2 Abramowitz, M., and I. Stegum, eds., 1964. Handbook of Mathematical Functions, Applied Mathematics Series SS, National Bureau of Standards, U.S. Government Printing Office, Washington, D.C. Also available from Dover Press.
OCR for page 187
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Resources Funding sources for the Mathematical and Computational Sciences Division (in millions of dollars) are as follows: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 4.1 3.8 Competence 0.1 0.1 STRS-Supercomputing 1.0 0.7 ATP 0.2 0.2 OA/NFG/CRADA 0.2 0.3 Total 5.6 5.1 Staffing for the Mathematical and Computational Sciences Division currently includes 30 full-time permanent positions, of which 28 are for technical professionals. There are also 15 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. The division's computer facilities seem quite adequate to support the workload, and the panel heard no complaints of any particular problems in maintenance and general support services. Last year, the panel was concerned that the location of MCSD in NIST North had a detrimental effect on the level of the division's ability to expand its collaboration with scientists in the main campus laboratories through informal encounters. However, the panel heard fewer complaints this year and suspects that there has been a certain amount of adaptation. Furthermore, the realities of the long-term realignment of the entire NIST campus suggest that patience will be its own reward and that the MCSD staff should continue to reach out to the campus laboratories as much as possible. The management and staff should not let new ventures be restricted by minor problems of geography. The MCSD has been through considerable transition in the past few years, first with the formation of ITL and the naming of a director who wanted to change some directions the division had taken in the past, and then with the subsequent retirement of their division chief. These events have led to morale problems in the past year. However, in the panel's view, the division is now past the low point and things are looking up for the division as a whole. There appears to be a fresh opportunity, with the naming of a new division chief, for the division to move forward with renewed spirit and new ideas. Discussions with candidates ought to be held with that thought in mind. Morale would also be improved by better communication to staff of ITL directions and their implications for MCSD personnel. It is clear to the panel that morale in the past year was adversely affected by the perception on the part of many in the division that they were given no clear sense of direction. Staff often thought they knew divisional priorities, only to be presented with different ones from management. The panel is not in a position to understand how these problems in communication came about, but they are real, and it will take a concerted effort on the part of the ITL director and all MCSD management to overcome them.
OCR for page 188
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Advanced Network Technologies Division The Advanced Network Technologies Division mission is to deliver measurement science to the communications industry. Despite the broad definition of the division's mission, all the division's projects are well justified. NIST can fill a unique role in standards by: Evaluating competing draft standards and extracting the technical differences among them. NIST is particularly effective here because it is seen as an “honest broker”; Providing freely available reference source code; Providing interoperability testing in a convenient and private manner; and Developing conformance tests. Each of the division's projects involves at least one of the above activities. Technical Merit and Appropriateness of Work The Advanced Network Technologies Division is performing state-of-the-art research in many areas. For example, the effort to develop a Java-based mechanism for multimedia collaboration among participants on different platforms is an exciting and important new technology. The division has also developed Web-based interoperability testing, which is a clever alternative to technology evaluation conferences. It reduces the expense of gathering and interconnecting independent implementations to test and allows testing in a risk-free private environment. This Web-based tester is currently being used to support Internet Security Protocol (IPsec) testing and will be expanded to provide services for other systems. IPsec is an important technology that has up to now been slow at being standardized and deployed, and the division's reference code will almost certainly speed up its deployment. The use of formal specifications by the High-Speed Network Technologies Group (in concert with Bell Laboratories) provides a precise specification of protocols and could lead to automatic generation of conformance tests. The Software Conformance Group is doing similar work, and collaboration may be in order. Although projects in internetworking technologies and multimedia seem appropriate, it is not certain that these efforts should focus totally on asynchronous transfer mode (ATM) technology. Backbones might be constructed in other ways, such as with pure fast Internet Protocol (IP) routers, or IP routers that achieve ATM speed through tag switching. Furthermore, IEEE Protocol 802.14 for cable television modems is attracting considerably less industry interest than it did last year and should be considered for phasing out. Impact of Programs The Advanced Network Technologies Division is active in the Internet Engineering Task Force (IETF), the ATM Forum, and the International Telecommunications Union, which are
OCR for page 189
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 currently the major standards-setting groups in networking. The division is also involved with the Cross-Industry Working Team and the International Multimedia Teleconferencing Consortium. Probably the most important protocol, and the one whose success the division has the most opportunity to influence, is IPsec. The IPsec reference code is clearly popular with industry; 30 companies have applied for the code thus far. Some are using it for reference, others for code base, and others for testing purposes. The division was a coauthor of the IPsec protocol specifications at the request of IETF. Similarly, the division's Web-based interoperability tester is also clearly of interest to industry. Although it is very new, and NIST is not necessarily notified when someone uses the tool, at least 14 companies are known to have tested against it. Another important area is IP quality of service, which needs NIST 's help. Reservation Protocol is assumed nonscalable, and many Internet service providers are refusing to deploy it. This protocol and its alternatives need to be studied by unbiased bodies like NIST. NIST can play a unique role in moving standards forward and getting them deployed. Standards bodies often get delayed, sometimes by years, because they are forced to compare competing complete proposals. A company may have a large financial and marketing stake in having their proposal chosen or individuals may have great ego invested in their proposal. It is invaluable to have an independent body like NIST separate out the individual subproblems from the various proposals and compare approaches for each subproblem. This leads to a fair process for arriving at consensus. The division has provided such services in the case of the hybrid fiber coaxial Media Access Control protocol, where they identified common portions of 18 separate proposals and synthesized a single coherent protocol. Once a standard has been adopted, NIST can play a major role in facilitating its deployment. The division is focusing its efforts on the right parts of this process, including making source code available for reference, providing code base or interoperability testing, creating conformance tests, creating interoperability tools, and hosting technology evaluation conferences. In addition to working with standards committees, the division has worked directly with industrial collaborators, including AT&T, Bay Networks, Cisco Systems, COMSAT, and IBM. The division also collaborates with universities and other government agencies, including the Defense Advanced Research Projects Agency (DARPA) and other NIST laboratories and divisions. Resources Funding sources for the Advanced Network Technologies Division (in millions of dollars) are as follows: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 3.8 3.3 ATP 1.1 0.3 OA/NFG/CRADA 0.7 1.8 Total 5.6 5.4
OCR for page 194
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 and methodology for evaluating speech-based document retrieval systems. This work is timely in view of the importance of retrieving both text and spoken information from large databases. The group collaborated with the Spoken Natural Language Processing Group to develop a state-of-the-art voice-actuated document-retrieval system. The face and fingerprint recognition work in the Visual Image Processing Group is very good and is leading to standards that are being adopted by law enforcement agencies. The customers, such as the Federal Bureau of Investigation, are providing serious support to NIST for this work. The division's programs in visualization and virtual reality do not yet show a coherent direction that points to significant progress. The Web usability testing program does not appear to provide much useful information and does not merit serious attention. What is really needed in this area are metrics for proper evaluation of multimodal interface technologies beyond mouse and keyboard. Impact of Programs The speech corpora developed by the division have helped the research community to make steady progress in improving the performance of speech recognition systems over the past 10 years. The results of these benchmark tests have helped the community to keep track of its progress. The most recent speech corpora, based on TV and broadcast news, have also helped the community to move towards real applications. The division's fingerprint recognition work had a strong impact on law enforcement work, making it possible to access fingerprint databases in a speedy manner. The face recognition work has resulted in more reliable means of identifying persons and in providing access to such data. Resources Funding sources for the Information Access and User Interfaces Division (in millions of dollars) are as follows: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 4.8 4.5 ATP 0.0 0.2 MEP 0.1 0.0 OA/NFG/CRADA 2.0 3.0 Total 6.9 7.7
OCR for page 195
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Staffing for the Information Access and User Interfaces Division currently includes 41 full-time permanent positions, of which 38 are for technical professionals. There are also 13 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. These resources are sufficient for effective execution of all the programs in the division. High Performance Systems and Services Division Mission The High Performance Systems and Services Division (HPSSD) enables effective application of high-performance computing and communications (HPCC) systems in support of NIST and its interactions with industry, academia, the federal government, and the public by (1) conducting research, development, and evaluation of innovative measurement and test methods, system architectures, and software technologies for improved scalability, functionality, flexibility, reliability, and economy for HPCC; (2) serving as a testbed for R&D in high-performance computing (HPC) and information technologies, and gaining experience in the deployment of these technologies and assessing their functional capabilities, interoperability, and operational characteristics; (3) serving as a responsive, effective mission-critical resource spanning computational, communication, mass storage, security, archival, and scientific visualization services; and (4) providing and managing state-of-the-art facilities that integrate and support an enterprise-wide heterogeneous information technology environment for NIST. With minor exceptions, the division programs are clearly in accord with the division's and NIST's missions. The program emphasis, in terms of budget, people, and effort, is strongly in the service areas supporting mission goals (3) and (4). There are smaller-scale efforts supporting the research mission (1) and the testing mission (2). This approach is appropriate considering the history and the scope of the division. Technical Merit and Appropriateness of Work This discussion of technical merit is organized according to the three areas of HPSSD program activity: service, research, and testing. Service. The service function dominates the division's work. The service functions of the HPSSD include responsibilities for telephone services, networking, and high-performance computing. Telephone. The planned telephone upgrades are necessary given the growth in usage on the site and the lack of support from the vendor of the legacy phone system. It is hoped that the planned acquisition will take into account data-networking needs and leave room for future expansion.
OCR for page 196
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Network. Planned capacity upgrades to the network are required to meet expected bandwidth needs for data communication, video teleconferencing, and multimedia. The panel encourages high bandwidth connection to the outside, including very high speed Backbone Network Series and future Internet 2 sites. The planned full-scale firewall is an excellent first step in implementing computer security for NIST. It is not a panacea for all security issues. Note particularly that the interaction of the phone system and the firewall needs to be considered carefully (e.g., unprotected modem access inside the firewall should not be allowed). High-Performance Computing. HPC support is a key service role for HPSSD. By enabling NIST researchers to make the best use of state-of-the-art resources, HPSSD enables other divisions to compete in their disciplines. To this end, it is essential that the HPSSD track the technological developments in HPC. HPSSD maintains a diverse stable of hardware resources. Some of these, heavily used in the past, are now antiquated and are no longer state of the art or cost-effective. For example, the Cray C-90 and Convex ought to be phased out. Alternatives for users who must have Cray vector access in the future might include leased time at the National Science Foundation, Department of Energy, or other centers. State-of-the-art resources such as the SP-2, the Origin 2000, and PC clusters should be scaled up to provide needed levels of service. Simultaneous exploration of clustered computers and distributed shared memory for problems not amenable to cluster computing should be encouraged. Further, considerably more capability is needed to remain competitive, let alone to be first class. To get a sense of scale, HPSSD currently maintains an eight-processor Origin 2000. In contrast, university computer centers often feature 32 to 64 processors and range up to as many as 192 processors. HPSSD has made strides in educating and helping the user community exploit HPC resources. The panel applauds their shift toward real collaboration with domain scientists rather than acting as mere programmers. This has accelerated the shift toward parallel computing at NIST. The panel encourages HPSSD to educate and train all orphaned Crazy and Convex users in the use of modern HPC equipment. This training will require additional resources. HPSSD-developed tools to aid in the use of HPC systems represent good support for the NIST community. The needs of NIST users are similar to those of other high-end computational scientists. The panel encourages HPSSD to maintain contact with recent large-scale HPC computational science efforts such as the National Science Foundation 's Partnerships for Advanced Computational Infrastructure, the Department of Energy's Accelerated Strategic Computing Initiative, and the Department of Defense's Defense Modernization Programs. Research. The research component of HPSSD is relatively small. It is not as comprehensive as might be desired but consists of various small projects with varying degrees of conformance to the division mission. For example, it is unclear that the electronic book (E-book), pattern recognition, and photonic interconnection for wireless communication projects fit within the division mission. The Photonic Interconnect project, in fact, seems a better fit to the Advanced Network Technologies Division. Work on the properties of flat-panel displays is marginally related to the mission and should at least be coordinated with the NIST Electronics and Electrical Engineering Laboratory and the NIST Physics Laboratory. However, research into PC clusters
OCR for page 197
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 for HPC is an exciting area being investigated throughout the world, and work in HPSSD is state of the art and appropriate to the division mission. Projects are at various stages of development although all seem to have clear time lines. Testing. The division's participation in development of a testbed for scalable computing over an interoperable message passing interface (IMPI) is very appropriate and helps move the industry toward truly flexible scalable computing. The division could provide a useful service to the broader community by testing and benchmarking HPC hardware and software; however, given the division's level of staffing, and the necessity to remain a neutral arbiter, the division's decision not to do so may be appropriate. Impact of Programs The panel interviewed several NIST users of the division's service programs who were very positive about services received. User response to the division has also been very positive. This was especially evident for the collaborative model the division uses in developing parallel applications and for services supporting administrative computing systems. The division's publication level has increased substantially, which improves the reputation and visibility of NIST in this discipline. Ongoing activities in HPC have not yet had a significant impact on that industry. There is a potential for future impact through the division's work on IMPI and clusters. Time synchronization and digital video disk (DVD) projects also represent technologies with potential industrial impact outside of the HPC industry. Also, a division staff member was recognized by the Video Electronics Standards Association for his leadership in standards issues relating to digital display interfaces. Resources Funding sources for the High Performance Systems and Services Division (in millions of dollars) are presented below: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 1.7 1.7 Competence 0.5 0.1 STRS-Supercomputing 8.3 9.8 ATP 0.5 0.5 OA/NFG/CRADA 0.8 0.4 Other Reimbursable 0.9 0.7 Agency Overhead 4.9 5.1 Total 17.6 18.3
OCR for page 198
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Staffing for the High Performance Systems and Services Division currently includes 71 full-time permanent positions, of which 51 are for technical professionals. There are also 17 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. In order for NIST to have first-class computational resources, more money will be necessary for capital expenditures on computers and networking. NIST management also needs to understand that traditional vector computing is not a cost-effective solution, either from the viewpoint of hardware maintenance or from the viewpoint of software migrated from the desktop. In addition, in the rapidly changing HPC environment, the procurement cycle must be reduced and costs amortized over a small number of years. A generation for HPC is 2 to 3 years. As with the whole of ITL, salary is a major problem in hiring new staff. As in other areas of ITL, flexibility and research opportunities offset the salary disincentives. However, on the service side, this effect is much reduced. Therefore there is a risk that it will become more and more difficult to maintain staff in service positions. Distributed Computing and Information Services Division Mission The mission of the Distributed Computing and Information Services Division (DCISD) is to provide an integrated suite of information technology resources and services that meets the needs of NIST staff in an efficient and effective manner and that provides a sound basis for meeting future needs and enabling innovative applications of information technology. The division provides software for scientific computing, support for local PCs and scientific workstations, electronic information dissemination services, and administrative and management applications support, directly supporting service functions and indirectly supporting the measurements, standards, and research functions of NIST. The division provides services and supports infrastructure to help the NIST staff, its collaborators, and its clients to address the measurement and standards needs of many industry sectors; therefore, its mission conforms to and helps to carry out the overall NIST mission to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards. Technical Merit and Appropriateness of Work The DCISD is involved in a number of activities to reduce the maintenance cost and improve the effectiveness and efficiency of NIST computing. Examples are unifying e-mail services, implementing a software checkout program, providing training on Microsoft Office Suite™, defining workstation security policy, negotiating site-wide license agreements, investigating Web training, and disseminating information via the Web. In addition to the continual service activities provided by DCISD, six projects are in progress as of the review. Although the Year 2000 compliance project, NT Enterprise Architecture, electronic funds transfer (EFT) disbursements, and similar projects are necessary for supporting NIST needs, collaborative projects such as EC, PKI implementation, Intranet streaming audio and video implementation, and
OCR for page 199
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 EFT draw DCISD personnel into research activities with other ITL divisions. Involving DCISD personnel in such a way is an inducement to the hiring and retention of good people. Impact of Programs The DCISD has carried out many upgrade projects at NIST effectively and efficiently. The division's support of a NIST-wide World Wide Web and its use to disseminate information on NIST programs is good. Feedback received by the panel from some internal customers was positive, although this sample may have been weighted toward customers who understood available technologies and the services they receive. There is a need for the division to conduct frequent customer surveys to obtain feedback on the effectiveness of the service portion of its organization. Because the division focuses on serving the IT needs of its NIST staff, its program has no direct impact on industry. However, the division could take an active role as a testbed for the IT industry, for example, advising the industry on the Year 2000 efforts and sharing its experience on scalability and reliability of NT clusters. Resources Funding sources for the Distributed Computing and Information Services Division (in millions of dollars) are as follows: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 0.4 0.5 STRS-Supercomputing 0.9 0.9 OA/NFG/CRADA 0.2 0.2 Other Reimbursable 0.5 0.6 Agency Overhead 5.7 7.0 Total 7.7 9.2 Staffing for the Distributed Computing and Information Services Division currently includes 62 full-time permanent positions, of which 57 are for technical professionals. There are also seven nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. Resources are probably the division's biggest challenge. The division is continually understaffed due to the competitive job market for the skills it needs and the uncompetitive salary that government pay scales restrict it to. As noted previously, the opportunity to spend some time working on collaborative research projects with other ITL divisions has been an inducement that partially compensates for salary inequities.
OCR for page 200
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 The division finds itself trying to keep pace with changes in technology while continuing to support old systems (both hardware and software) for users who do not see the need to upgrade. The recent implementation of Microsoft Office as the standard office software at NIST has occurred in principle, but not entirely in practice. The division faces tough decisions on which systems it should support centrally out of overhead funds and which should be left to individual operating units to support directly. The continued upgrade of all systems every 3 to 4 years would reduce the complexity of maintenance and improve the effectiveness of the division's staff, but the appropriate funding for the upgrade must be allocated. It is hard to imagine a state-of-the-art organization still using 386 and 486 machines, but that is the case at NIST. An active division effort to educate and make recommendations to the NIST community on hardware and software upgrades may help bring about acceptance of and demand for newer technologies. Software Diagnostics and Conformance Testing Division Mission The mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; participate with industry in the development of forward-looking standards; and lead efforts for conformance testing, even at the early stages of development of standards. Technical Merit and Appropriateness of Work The division's work in designing conformance tests is state of the art and clearly relevant to the division's mission. The division's work in development of virtual reality modeling language (VRML) test suites continues, with much progress made in the past year. The division has now released tests that cover 60 percent of VRML specifications and has completed development of a reference parser and scene graph generator for VRML. To this point, the division has focused only on static testing but is planning work on testing dynamic VRML worlds. There may be a potential to accelerate and improve this work through collaborations with the Information Access and User Interfaces Division, which has efforts in both VRML and virtual worlds. Work on Java test suites has only recently been initiated, despite the widespread use of the language in industry. The division has demonstrated inconsistencies in Java virtual machine (VM) implementations, and is developing a transparent VM that allows viewing of VM components during execution. This work is quite relevant to industrial users, but it is not clear that NIST has a unique role to play, as industry is already quite active in this area. The division has exciting interactions with researchers from the SED on software testing by statistical methods. Although this project is based on a concept developed almost 20 years ago, the practical realization of this concept will be highly useful to industry. The enthusiasm of the participating staff from both divisions demonstrates the fruitfulness of the collaboration for all partners.
OCR for page 201
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Continuing work in role-based access control, in collaboration with the Computer Security Division, is of good technical quality but lacks a sense of direction. The concept is very useful for industry, and program management might consider strategies for increasing the visibility and impact of the efforts. The division's related work in electronic commerce is innovative both in the attempt to use an instruction management system and in the design of an architecture for electronic commerce. Designing for future standards based on this architecture could be very important. Again, a clear focus is necessary for this work to have an impact. A Competence project to develop automated software test generation using formal methods is a very ambitious project but worth the preliminary efforts being expended. Smartcard vendors have already expressed interest in using such a method to generate tests for smartcard verification. Resources Funding sources for the Software Diagnostics and Conformance Testing Division (in millions of dollars) are presented below: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 5.2 4.4 Competence 0.0 0.5 ATP 0.3 0.3 OA/NFG/CRADA 1.5 1.9 Other Reimbursable 0.4 0.3 Total 7.4 7.4 Staffing for the Software Diagnostics and Conformance Testing Division includes 41 full-time permanent positions, of which 34 are for technical professionals. There are also 10 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. The panel was impressed with the high level of enthusiasm displayed by the staff of the division. The staff members seem to have a good sense of the direction that the division is taking and are engaged in the division mission. Statistical Engineering Division Mission The SED states that its mission is to catalyze experimentation, enhance research, and improve communication of results by working collaboratively with, and developing effective statistical methods for, NIST scientists and partners in industry. The role of this division is now articulated in the ITL mission statement. In support of its mission, ITL provides technical
OCR for page 202
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 leadership and collaborative research in critical infrastructure technologies that promote better development and the use of information technology, in addition to high-quality services and supporting infrastructure to help the NIST staff, its collaborators, and its clients address the measurement and standard needs of many industry sectors. Technical Merit and Appropriateness of Work The SED continues to play an integral and successful role in the core business of NIST, with high-quality work and collaborations ongoing throughout the NIST laboratories. It also appears to be unusual for a division of this size to have three NIST Competence funding awards. The division's work with SEMATECH on an online Internet Engineering Statistics Handbook for scientists and engineers in industry has progressed well and will be a highly useful and important product. All chapters are expected to be completed in the coming year. Another important and useful project is the award-winning Statistical Reference Datasets. The purpose of this Web-based service, now in place, is to improve the accuracy of statistical software by providing reference data sets with certified computational results that enable objective evaluation of statistical software. The division continues to pursue efforts to more efficiently process the analysis of the uncertainty associated with the measured quantities and characteristics certified in NIST Standard Reference Materials. They have increased the number of their analyses significantly without a corresponding increase in resources. It is not clear if much more efficiency is possible at this point. The division's program of short courses and short tutorials for the NIST community serves to provide some contact with other scientists and engineers at NIST in spite of the SED's isolated location at NIST North. Other laboratories and other ITL divisions report favorably on their impact. The division's annual report3 is an impressive catalog of research activities and collaborations with multiple NIST laboratories, including other divisions in ITL. The percentage of effort devoted to projects with other ITL divisions has grown to 10 percent, a significant increase over last year. The panel is pleased by this increase and sees the possibility for still more mutually productive interactions with other ITL divisions. A large number of publications in high-quality journals is strong evidence of the technical merit of the work. Impact of Programs The StRD and the Engineering Statistics Handbook are too recently available on the World Wide Web to evaluate their impact, but counting the number of hits would allow for some estimate of their use. The large number of publications by the division is an indication of the dissemination of their work. Much of the industrial impact is indirect through collaborations with other divisions and laboratories in NIST. 3 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Report of Activities: Statistical Engineering Division, NIST internal document, National Institute of Standards and Technology, Gaithersburg, Md., 1998.
OCR for page 203
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 Resources Funding sources for the Statistical Engineering Division (in millions of dollars) are as follows: Fiscal Year 1997 Fiscal Year 1998 (estimated) NIST-STRS, excluding Competence 2.6 2.4 Competence 0.1 0.3 STRS-Supercomputing 0.5 0.4 OA/NFG/CRADA 0.0 0.2 Other Reimbursable 0.1 0.0 Total 3.3 3.3 Staffing for the Statistical Engineering Division currently includes 21 full-time permanent positions, of which 19 are for technical professionals. There are also 10 nonpermanent and supplemental personnel, such as postdoctoral fellows and part-time workers. The demands on SED by other laboratory personnel for collaborations and consulting need to be balanced with the effort to increase interactions with other ITL divisions. The unfilled positions and no-growth budget have affected the ability of SED staff to respond to the full demand for its expertise. The development of the Advanced Measurement Center for Information Technology, an ITL collaboration with the University of Maryland, is a promising avenue for reinforcing SED and other ITL division interactions and for bringing new external scientific expertise into “software metrology.” SED should take full advantage of this opportunity to supplement staffing by these interactions. MAJOR OBSERVATIONS The panel presents the following major observations. ITL and NIST management should explicitly recognize and support the varied roles of the divisions within ITL and measure their performance and allocate their resources with a clear eye to the differing work profiles within the divisions. One size does not fit all. New policies that require MCSD and SED to seek nonoverhead resources should be reassessed carefully; this is not a step to be taken lightly. The collaborative and research functions of the Statistical Engineering and Mathematical and Computational Sciences Divisions still need to be more fully integrated into the ITL's activities and mission. Information technology is a dynamic area with an extremely high demand for skilled people in all roles. NIST must work toward increased compensation for such employees and must find creative ways to attract and retain such employees.
OCR for page 204
An Assessment of the NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY MEASUREMENT AND STANDARDS LABORATORIES Fiscal Year 1998 NIST must find ways to express appreciation and give credit to those in supporting roles or whose collaborations are behind the scenes. The ITL would benefit from more detailed tracking of the results and consequences of its technical activities. Maintaining old technologies such as the Cray supercomputer or outdated software on older desktop machines is not resource efficient. Equipment should be upgraded, and only common hardware and software should be supported out of overhead funds. The departure of the SED chief in the past year has led to the appointment of an acting chief. Filling this position rapidly with a permanent division chief is essential to staff morale and necessary to accelerate the growth of SED interactions with other ITL divisions.
Representative terms from entire chapter: