Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 187
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Chapter 8 Information Technology Laboratory
OCR for page 188
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 PANEL MEMBERS Brian W. Kernighan, Lucent Technologies, Co-Chair Richard E. Nance, Virginia Polytechnic Institute and State University, Co-Chair Bishnu S. Atal, AT&T Laboratories Research Matthew Bishop, University of California, Davis Linda Branagan, Construct Internet Design Jaime Carbonell, Carnegie Mellon University Josephine Cheng, IBM Santa Teresa Laboratory John R. Gilbert, Xerox Palo Alto Research Center Roscoe C. Giles, Boston University Andrew S. Grimshaw, University of Virginia Thomas P. Kehler, Adaptivity Jon R. Kettenring, Bellcore John W. McCredie, University of California, Berkeley Vijayan N. Nair, University of Michigan Lawrence O'Gorman, Veridicom, Inc. Thomas Parenty, Sybase, Inc. Radia Perlman, Sun Microsystems James L. Phillips, Boeing Shared Services Group K.K. Ramakrishnan, AT&T Laboratories Research Ahmed N. Tantawy, IBM Corporation Stephanie M. White, System World, Inc. Eddie L. Zeitler, Charles Schwab & Company Submitted for the panel by its Co-Chairs, Brian W. Kernighan and Richard E. Nance, this assessment of the fiscal year 1999 activities of the Information Technology Laboratory is based on a site visit by the panel on February 16–17, 1999, in Gaithersburg, Md., and documents provided by the laboratory.
OCR for page 189
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 LABORATORY-LEVEL REVIEW Laboratory Mission According to laboratory documentation, the mission of the Information Technology Laboratory (ITL) is to promote the U.S. economy and public welfare by providing technical leadership for the nation's measurement and standards infrastructure for information technology (IT). The laboratory carries out this mission by working with industry, research, and government organizations to develop and demonstrate tests, test methods, reference data, proof of concept implementations, and other infrastructure technologies that are needed by U.S. industry to produce information technology systems that are usable, secure, scalable, and interoperable. The ITL's role includes four components: research, measurement, standards, and service. This combination of roles is unique among the laboratories at NIST, and the variety among these activities makes ITL especially heterogeneous. Each division within the laboratory has an important role to play, but the relative emphasis on each of the four components varies greatly from division to division. Progress on integrating the eight divisions of ITL into one unified, coherent laboratory continues, and most of the divisions seem to have moved past the transitional difficulties that the panel has commented on over the past several years. In particular, the Mathematical and Computational Sciences Division has successfully addressed many of the issues related to its role within ITL; this process has been helped significantly by the appointment of a permanent division chief. Progress has been made on defining the range of the collaborative research functions of the Statistical Engineering Division into the ITL, but this task is not yet complete and will continue to require care and attention on the part of management at all levels of NIST. Statistics plays a vital role in NIST activities, both through the traditional approach of collaborating with scientists in other NIST laboratories and through the newer pathways of contributing to studies of information technology issues within ITL and working directly with industry. It is important to manage all of these interactions carefully and to understand the value of the statisticians' contributions and the need for a transition that takes into account both work programs and the individuals involved. Technical Merit and Appropriateness of Work The quality of the programs in the laboratory is discussed in detail in each divisional report. With few exceptions, also detailed in the divisional reports, the programs are appropriate and well aligned with the laboratory and divisional missions. The ITL's planning process continues to improve. This year, the panel observed several examples of projects that were undertaken with a clear definition of their intended value and NIST's likely contribution, that were carried through to a suitable degree of completion, and that were terminated or scaled back gracefully when NIST's participation was no longer vital. Instances include the Software Diagnostics and Conformance Testing Division's work on role-based access control (RBAC) and virtual reality modeling language (VRML) and the Advanced Network Technologies Division's asynchronous transfer mode (ATM) testing efforts. In several
OCR for page 190
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 cases, the experiences and methodologies from completed projects will be applicable in future activities; for example, the skills and approaches learned during work on VRML will help in the new project on extensible markup language (XML). ITL has identified several new initiatives, and fiscal support for these new undertakings will come from reprogramming, in which as much as 20 percent of laboratory funding is shifted away from current work and into new projects. In general, the panel viewed this redistribution of support as a good method of encouraging laboratory staff to build a changing mix of activities, to develop innovative proposals, and to explore new opportunities. The largest of these new initiatives is laboratory-wide work on pervasive computing, which is a broad area containing a wide variety of topics potentially suitable for study. Which of these many topics NIST will work on still needs to be more clearly defined, and the goals of the initiative need more focus. The panel believes that ITL should concentrate on topics where NIST possesses special expertise or has a unique role to play. In general, such criteria would imply work on early standardization activities like helping to develop guidelines, coordinating consortium activities, building reference implementations, establishing test criteria and methods, and other similar projects. Since so many companies and universities are active in the general field of pervasive computing, ITL risks being seduced by intriguing research problems that could divert focus away from more appropriate NIST activities. For example, ITL may not be uniquely qualified to develop yet another protocol in this field, but the laboratory could provide significant value to the community by developing tests that verify or demonstrate deficiencies in existing protocols. Another proposed activity is the Knowledge Exchange initiative, which would have the broad goal of helping create a value system for intellectual property. The panel believed that this activity needed significantly more clarity and focus; the projects that would be undertaken and the relationship of the work to the ITL or NIST mission were not at all clear. Industrial Impact As noted in last year's report, ITL's efforts to reach out to the industrial community continue to improve, and the resulting partnerships have been productive for all parties. As the laboratory's portfolio of programs changes and as industrial priorities evolve, NIST staff will have to continually reevaluate the relationships with outside institutions in order to maintain a strong connection to the constituencies that benefit from technological and metrological advances made in the ITL. ITL has numerous projects that directly affect U.S. industries. Perhaps the most visible is the Advanced Encryption Standard (AES), which seems to be well on track to provide a worldwide replacement standard for the Data Encryption Standard, which was first approved in 1977. There are also other security-related activities that are already important and influential or hold the prospect of being so. ITL is involved in a variety of industry consortia and groups; these activities have been an especially effective way to positively influence industry at an early stage in the evolution of a particular technical field. Successful examples include biometrics, Java Real-time and Java Numerics, speech corpora and testing, and text recognition. For very young or particularly contentious technologies in which industry consortia have not yet crystallized, NIST has utilized its reputation as a neutral party to host workshops and instigate industry-wide discussions of
OCR for page 191
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 standards. One example of this technique in action is the Electronic Book workshop held in February of 1999. This meeting was a particularly effective and highly visible forum that brought together a wide variety of disparate groups. In addition to its interactions with industry communities, ITL continues to have a positive impact on other laboratories within NIST, where scientists rely on ITL staff for research collaborations and technical services. The Mathematical and Computational Sciences Division and the Statistical Engineering Division have a long history of supporting work in other NIST laboratories, and they continue to perform crucial services in areas such as modeling and validation of standards activities. As ITL appropriately seeks to refocus some of the mathematical and statistical work in these divisions onto information technology issues under study in other ITL divisions, it is important not to lose sight of the value of the traditional work as well. A final element of the impact of the ITL is its provision of vital services to the entire NIST community; these services include networking, high-performance computing, computer support for desktop and workstation machines, the telephone system, and a host of other infrastructure activities. The value of these activities may not be fully appreciated, or the level of appreciation across NIST may not be accurately communicated to the staff working in these areas. The panel stresses that this part of ITL is very important and efforts to emphasize the value of this work should continue.
OCR for page 192
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Laboratory Resources Funding sources1 for the Information Technology Laboratory (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 31.4 31.5 Competence 1.0 1.5 STRS-Supercomputing 11.9 12.0 ATP 1.8 0.9 OA/NFG/CRADA 10.6 12.8 Other Reimbursable 1.5 0.6 Agency Overhead 12.0 13.6 Total 70.2 72.9 The panel is pleased that the laboratory's dependence on other agency (OA) funding is holding at a comfortable level of under 20 percent. However, at the divisional level, panel members heard several independent comments that money was very tight and that outside funding was being actively pursued to make up for real or anticipated shortfalls. Such activities should be monitored carefully, lest OA funding begin to creep up again to higher levels. As of January 1999, staffing for the Information Technology Laboratory included 381 full-time permanent positions, of which 314 were for technical professionals. There were also 91 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The present staff are strong and qualified, but the difficulties in hiring new personnel that were noted in last year's report remain a serious issue. Over the past year, there have been several significant losses in both professional and support staff, and the current intense market demand for information technology professionals makes finding replacements difficult.2 The joint institute with the University of Maryland was expected to attract a series of potential 1 The NIST Measurement and Standards Laboratories funding comes from a variety of sources. The laboratories receive appropriations from Congress, known as Scientific and Technical Research and Services (STRS) funding. Competence funding also comes from NIST's congressional appropriations, but it is allotted by the NIST director's office in multiyear grants for projects that advance NIST's capabilities in new and emerging areas of measurement science. Advanced Technology Program (ATP) funding reflects support from NIST's ATP for work done at the NIST laboratories in collaboration with or in support of ATP projects. Manufacturing Extension Partnership (MEP) funding reflects support from NIST's MEP for work done at the NIST laboratories in collaboration with or support of MEP activities. Funding to support production of Standard Reference Materials (SRMs) is tied to the use of such products and is classified as Measurement Services. NIST laboratories also receive funding through grants or contracts from other government agencies (OA), from nonfederal government (NFG) agencies, and from industry in the form of Cooperative Research and Development Agreements (CRADAs). All other laboratory funding, including that for Calibration Services, is grouped under Other Reimbursable. In addition to performing research and providing calibrations and reference data, the Information Technology Laboratory also provides services to the other NIST laboratories. This work is supported through STRS funding designated specifically for the supercomputing facilities and through NIST overhead. 2 For graduating PhDs in this area, NIST can only offer starting salaries that are substantially below the level offered by private businesses.
OCR for page 193
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 collaborators on short-term visits and to provide an infusion of new, young talent through graduate students on postdoctoral appointments. In addition, it was hoped that the cooperative arrangement would supply some alternative funding arrangements. However, the agreement between NIST and the university has still not been finalized, and the delay is having a negative effect on hiring. Overall, morale appears to be quite high; in some divisions, the improvement over previous years is significant. The panel co-chairs met with employees who were new to NIST and separately with long-time employees. Although both groups expressed some frustrations with the space available, with the geographical separation of some divisions at NIST North, and particularly with the Department of Commerce legal department, all of the employees seemed uniformly pleased to be working at NIST. Some factors that contributed to the staff's satisfaction were the quality of equipment available for their research and the access NIST employees have to experts at other government agencies, at universities, and in industry. Staff members who had worked previously in industrial settings felt that in ITL, the communication between management and staff was much better than at most companies, and this higher level of clarity about plans and goals both enabled staff to plan ahead about their projects and gave them the freedom to choose their own approaches to tackling the key issues in their work. The NIST director's initiative to institute management training and preparation for staff throughout NIST addresses a concern that is by no means unique to ITL: There are too few people who are ready for and interested in assuming a management role, and as a result, managerial departures can leave a significant vacuum. DIVISIONAL REVIEWS Mathematical and Computational Sciences Division Mission According to division documentation, the mission of the Mathematical and Computational Sciences Division is to provide technical leadership within NIST in modern analytical and computational methods for solving scientific problems of interest to U.S. industry. The division focuses on the development and analysis of theoretical descriptions of phenomena (mathematical modeling), the design and analysis of the requisite computational methods and experiments, the transformation of these methods into efficient numerical algorithms for high-performance computers, the implementation of these methods in high-quality mathematical software, and the distribution of this software to NIST and industry partners. The panel finds that the division programs are in excellent conformance with the division mission. Projects such as Micromagnetic Modeling and Optimal Signal Sets for Wireless Communication address applied mathematics issues directly relevant to high-technology industries important to today's U.S. economy. Other projects, such as the Digital Library of Mathematical Functions and Java Numerics, are aimed at developing important enhancements in the infrastructure for modern scientific computing. These two projects, as well as the work on Optimal Signal Sets, also directly support the ITL mission of providing technical leadership for the nation's measurement and standards infrastructure for information technology.
OCR for page 194
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 A number of projects involve collaborative work with other NIST laboratories, and division staff provide vital support to activities throughout NIST. Examples of such projects include modeling activities in acoustic emissions for nondestructive testing with the Building and Fire Research Laboratory; in alloy solidification processes and high-speed metal-cutting with the Materials Science and Engineering Laboratory; and in optical reflectance of surfaces with the Building and Fire Research, Physics, and Manufacturing Engineering Laboratories. Indeed, the presence of a highly qualified and professional group of mathematicians elevates the scientific potential of the entire NIST enterprise and should continue to be viewed as a fundamental part of the scientific infrastructure at NIST. Although the Mathematical and Computational Sciences Division has several key projects that directly support the ITL mission, the panel believes that more technical interactions between staff from this division and other personnel within ITL would strengthen the instances of supportive and collaborative work. Colloquia or project reviews in which speakers from across ITL focus their presentations for an audience of nonspecialists would facilitate such interactions. Technical Merit and Appropriateness of Work The work of the staff in the Mathematical and Computational Sciences Division is of high quality, and many key projects define the state of the art in computational and applied mathematics. Moreover, the engagement of prominent research mathematicians from outside NIST as part-time employees or consultants helps assure strong links to the broader research community. One example of the striking benefits of this approach can be seen in the Digital Library of Mathematical Functions project, in which nationally known experts are contributing as associate editors on topics in their area of specialty. The quality of the expertise present and the work done in the division is also evident through the recognition received by division personnel, such as receipt of the Department of Commerce Bronze Medal, the Presidential Early Career Award for Scientists and Engineers (the second such award for Mathematical and Computational Sciences Division personnel in 3 years), and the external publications, professional leadership work, and other professional activities by a number of division personnel. As noted above, the panel finds the work of the division to be very appropriate in the context of the division, laboratory, and NIST missions. The Java Numerics project (new in 1998) is an excellent example of what NIST should be doing in the age of information technology. NIST's role as neutral arbiter is built in part on the external reputation of the staff, and therefore NIST's leadership efforts are supported by remarkably strong industry buy-in. Other divisional projects are focused on more traditional problems but are no less appropriate activities for NIST. For example, the work on computing the three-dimensional dimer constant solves a problem that has been of interest to physicists for 60 years, as this constant can be used to compute thermodynamic properties, such as the specific heat of a material, from first principles. The Digital Library of Mathematical Functions project, begun in 1997, is also an exciting and appropriate project. The goal of this work is to construct a new Web-based mathematical reference source for special functions and their applications by providing the graphs, tables of numerical values, and mathematical formulations that are needed by scientists in many disciplines. However, significant resources need to be identified for the project to proceed in a timely fashion. The panel is concerned that without such resources, the project
OCR for page 195
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 could stagnate and changes in Web technology could make it difficult to efficiently complete the project without significant re-work along the way. In the Statistical Engineering Division, a similar Web-based effort is under way to produce an updated Web version of the resource known as the Handbook of Engineering Statistics.3 As currently developing, this Web publication has many well-designed interactive features. Although the content and customers of the Digital Library and the Handbook are quite different, both projects may benefit by looking at useful architectural features and layouts in the other. Moreover, where appropriate, links should be built between the Library and the Handbook. Although impressed by the technical content of the Digital Library effort, the panel also wishes to highlight the planning that went into this project, as it has a particularly well-defined end product and plan for completion. Overall, program planning within Mathematical and Computational Sciences Division programs should be strengthened. In general, clear directions are articulated for each project; however, goals and milestones against which progress and performance can be evaluated are not evident. The panel recognizes that budget uncertainties can make such project planning difficult, but nevertheless, stronger efforts should be made in this area. Impact of Programs Overall, the panel sees evidence that the work done in the Mathematical and Computational Sciences Division has had a strong positive impact on industry and on the scientific community at large. However, the variety of interactions that occurs both within and outside NIST make it difficult to determine the best ways to quantitatively measure the impact of the division's projects. The panel describes below several examples of the important work being done in the Mathematical and Computational Sciences Division and the actual and potential effects observed by the panel. The division has for several years been a pioneer in using the World Wide Web to disseminate its work. The Guide to Available Mathematical Software (GAMS) is one of NIST's most-linked-to sites4; Matrix Market is an example of how the Web can be used to disseminate SRMs; and the Digital Library of Mathematical Functions is an ambitious project in digital networked documents. Moreover, the division's practice of making its mathematical modeling codes, such as the Object Oriented MicroMagnetic Framework and Object Oriented Finite Element Software available to the public is a good policy, although it is hard to measure the actual impact such code-sharing has on the community. During the past year the division has increased its outreach to other ITL divisions and across NIST. An example of the promising results is the new program on Optimal Signal Sets, a collaboration with staff from the Advanced Networking Division in the ITL and from NIST's Electronics and Electrical Engineering Laboratory. The personnel involved are combining forces to solve a problem of importance to industry. The panel believes that it is essential to cast a wide net to catch such hard-to-predict winning connections; more forums in which staff from divisions 3 Natrella, M.G., Experimental Statistics, NBS Handbook 91, U.S. Government Printing Office, Washington, D.C., 1963, reprinted October 1966 with corrections. 4 GAMS serves roughly 12,000 users per month (where the activity of a given user may generate many separate hits to the Web site).
OCR for page 196
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 across ITL can learn of each others' skills and needs will assist in finding the key collaborative projects. The ways in which Mathematical and Computational Sciences Division projects have an impact on industry fall into two broad categories. The first type takes advantage of direct interactions with industry. This occurs either through collaborations on the evolution of a project, as in the Java Numerics work, or through the production of tools used in industry, such as GAMS and Matrix Market, both available on the Web. The second way in which divisional personnel affect industry is through their vital collaborations in mathematical modeling and scientific computing on projects based in other NIST laboratories. Some examples include work on machining processes in the Manufacturing Engineering Laboratory and the Optoelectronic Modeling project in the Electronics and Electrical Engineering Laboratory. Contributions and impacts from such work are very important to NIST and to fulfilling NIST's mission. NIST is holding a series of workshops, “Being the Best in the World,” and the Mathematical and Computational Sciences Division hosted one on modeling and simulation in June 1998. It is more difficult to define Best in the World in this broad area than in more specific domains of metrology like mass and temperature, but the June workshop did identify NIST's valuable role in developing and disseminating software, as well as in using modeling and simulation to support measurements and standards in other disciplines. Resources Funding sources for the Mathematical and Computational Sciences Division (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 4.3 3.1 Competence 0.1 0.2 STRS-Supercomputing 0.6 0.6 ATP 0.2 0.1 OA/NFG/CRADA 0.3 0.7 Total 5.5 4.7 The division has now achieved its goal of obtaining 20 percent of its funding from outside the division. This transition has been painful, and seeking out opportunities and preparing proposals have cost the staff a great deal of time and effort. Currently, most of the external support is due to a large number of small projects. The division expects that as external funding levels off at 20 percent, the time spent on finding and winning this support will decrease. Although this may be the case, management should monitor the progress in this direction carefully. As of January 1999, staffing for the Mathematical and Computational Sciences Division included 30 full-time permanent positions, of which 27 were for technical professionals. There were also 11 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.
OCR for page 197
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 This is one of the smallest divisions in the ITL, and the panel is concerned that staff attrition could drop the division below critical mass. One fiscal year 1998 departure has been a particular blow due to the resulting loss in internal leadership, technical expertise, and external leadership and visibility. Overall, the division has lost a number of people in both Gaithersburg and Boulder over the past several years, and it is important that resources be devoted to replacing key personnel. Nonetheless, the panel recognizes that moving the Compression Algorithms Group out of the Mathematical and Computational Sciences Division was a wise decision, in light of the group's small size and specialized interests. As the division begins to hire new people, the panel recommends that the Optimization and Computational Geometry Group be enhanced to provide discrete mathematics (and discrete computational methods) expertise and support for projects in information technology. Building a coherent, useful capability within the large field of discrete mathematics will be a challenge, and the development of a viable small- to medium-sized group will require strong internal synergy within the division and a clear understanding of the expertise needed to provide support in the specific fields important to ITL. One approach that may be practical is to hire postdoctoral research associates in discrete mathematics, as the panel believes that the prospect of attracting very high quality PhDs in this field is more realistic than in other areas of information technology. Morale continues to improve in the Mathematical and Computational Sciences Division. The efforts of NIST and ITL management to communicate directly with division staff and to reassure them of the importance of the division's continuing role both within ITL and across NIST have had a very positive impact. The appointment of a permanent and highly respected division chief has also boosted morale and can clearly be seen to be providing needed stability for the division. Although much progress has been made, the panel emphasizes that communication throughout the management chain should continue to be a high priority. Advanced Network Technologies Division Mission According to division documentation, the mission of the Advanced Network Technologies Division is to provide the networking industry with the best in test and measurement technology. This is an appropriate mission statement, which accurately reflects the NIST and laboratory missions in the context of the technologies relevant to this division's work. However, the panel was somewhat concerned that not all of the projects under way in this division were completely in conformance with this mission. Occasionally, research work seemed to be started only because a staff member was interested in a particular field rather than because the potential benefits of work in that area had been evaluated in light of NIST's special niche in measurement technology. A symptom of this problem is that sometimes the goals and objectives of projects are not clearly articulated; a contributing factor could be that the process through which projects are selected did not appear to the panel to be clearly defined.
OCR for page 208
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 appropriately. Work on extending and standardizing the firewall is a key activity that deserves management attention throughout NIST. The NIST systems are vulnerable to intrusion and damage until operations are brought behind a firewall. Impact of Programs Admirable progress was observed on the development of explicit and effective connections between division programs and industry needs. The ISIS project hosted a very successful workshop on electronic books that brought together industry leaders and highlighted NIST 's ability to create standards in new areas. Local high school students work as programmers at the laboratory on this project, and a CNN feature on these students provided significant positive public recognition for NIST. The new DASE project started in response to industry's requests for software standards for digital television applications. Indeed, the industry has high expectations for the value of NIST 's work in this area, and the resulting request for a very fast timetable may challenge the division's resources. Other examples of impact include the division's participation in NIST's work with the Biometrics Consortium and the collaborations between High Performance Systems and Services Division staff and personnel from other ITL divisions. The work done in migrating NIST users from the Cray to more powerful parallel and distributed computing resources has been very successful and promises to provide long-term improvements in high-performance computing for scientists throughout NIST. Resources Funding sources for the High Performance Systems and Services Division (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 2.4 2.4 Competence 0.1 0.0 STRS-Supercomputing 9.5 10.0 ATP 0.5 0.1 OA/NFG/CRADA 0.4 0.3 Other Reimbursable 0.7 0.0 Agency Overhead 5.3 6.1 Total 18.9 18.9 As of January 1999, staffing for the High Performance Systems and Services Division included 75 full-time permanent positions, of which 55 were for technical professionals. There were also 22 nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers.
OCR for page 209
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 It is important that the laboratory maintain an adequate level of support for high-performance computing as a service to NIST. With the Cray being phased out in early 1999, the division must be able to explore and deploy parallel systems (either shared memory or distributed or clusters) at a scale that can support the increasing needs of the laboratories. In the research area, successful coupling of division programs to fast-moving information technology industries (as occurred in the DASE project) may require flexible increments in funding in order to work on the fast time scales required. Distributed Computing and Information Services Division Mission According to division documentation, the mission of the Distributed Computing and Information Services Division is to provide the information technology resources, supporting infrastructure, applied research, and assistance to NIST staff, collaborators, and clients for application in the conduct of scientific and engineering studies, the development of administrative applications, and the broad dissemination of information. The division provides software for scientific computing, support for local PCs and scientific workstations, electronic information dissemination services, and administrative and management applications support. An integral component of the NIST program strategy is to leverage information technology services to support its overall mission to promote U.S. economic growth by working with industry to develop and apply technology, measurements, and standards. As a primary provider of information technology support to all of NIST, the division conforms to, and helps to carry out, this important mission. Technical Merit and Appropriateness of Work The panel observed that the overall breadth and scope of the Distributed Computing and Information Services Division programs and plans appear to match NIST's needs in information technology support. The division is involved in several activities designed to reduce maintenance costs and hence improve effectiveness and efficiency in the NIST computing environment. Examples of the responsibilities of this division include providing electronic mail and calendaring services, Usenet news, standard software packages, file services, and workstation and domain administration. Like utilities, these services are available 24 hours per day, 7 days per week, and they are heavily used by the NIST community. For example, in 1998 the software checkout service in Gaithersburg was accessed more than 30,000 times and the one in Boulder more than 26,000 times. In addition, the division hosts several internal and external Web and Anonymous FTP services for NIST departments. Division staff provide a secure Web architecture and consulting and training services. The division is also charged with supporting several significant administrative applications such as the software systems used for financial, personnel, and procurement management. As part of this responsibility, the organization has completed its year 2000 compliance work and tested more than 100 NIST administrative systems. Currently staff
OCR for page 210
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 members are involved in the implementation of the property system module of the Department of Commerce Administrative Management System at NIST. The PC Support Group provides software support, end-user training, and maintenance and upgrade services for NIST hardware. Group staff also coordinate a help-desk facility and manage several site-wide licensing agreements. To monitor the quality of work in this group, staff members developed an automated customer satisfaction survey. There have been more than 1,500 responses to this survey instrument, and users have expressed a high degree of satisfaction with the services they have received. This group also assisted more than 500 users in checking and testing their local PCs for year 2000 readiness. New programs are planned to provide increased support for Windows NT, Linux, and PC clusters and will meet the needs of customers distributed throughout NIST. Contributions to improving the security of the computing environment include upgrading operating systems, implementing a Public Key Infrastructure for NIST staff, installing intrusion countermeasures and detection software, and installing Web access to forms with digital signatures. The plan to apply the Baldrige Quality Criteria to the work of one group in this division is a positive organizational development step. The panel received a copy of the ITL Information Technology Services Plan for NIST.5 This document lays out an excellent overall strategy and plan for providing information technology services to the whole of NIST and describes the integration of the programs of the High Performance Systems and Services Division and those of the Distributed Computing and Information Services Division. However, both in a preliminary visit to the Distributed Computing and Information Services Division and in the full panel visit, there was little or no evidence of collaborative planning between the divisions, and no references were made to this excellent strategic plan. The panel suggests that in the future the Distributed Computing and Information Services Division should relate reports about its operational programs and plans to this coordinated strategic planning document. Impact of Programs As noted above, the Distributed Computing and Information Services Division provides a wide range of important support services to staff throughout all of NIST. The results of the automated customer service satisfaction survey indicate that these services are well received. Further efforts are under way to determine other methods through which users could be involved in additional service-related measurements. The active involvement of division staff in educating and advising the NIST community on hardware and software upgrades seems highly desirable. Given that NIST has stated its strategic dependence on a well-functioning, state-of-the-art information technology environment, the Distributed Computing and Information Services Division's role in making this vision a reality is of central importance. Through their support of personnel throughout NIST, staff in this division have an indirect impact on industry. 5 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Report to ITL Assessment Panel January 1999, National Institute of Standards and Technology, Gaithersburg, Md., Appendix E, pp. 57-116.
OCR for page 211
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Resources Funding sources for the Distributed Computing and Information Services Division (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 0.6 0.5 STRS-Supercomputing 0.9 0.8 OA/NFG/CRADA 0.1 0.2 Other Reimbursable 0.6 0.6 Agency Overhead 6.3 7.0 Total 8.5 9.1 As of January 1999, staffing for the Distributed Computing and Information Services Division included 72 full-time permanent positions, of which 65 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The division is responsible for PC hardware and software support, Web server maintenance, e-mail programs and management, large-scale administrative software, and a variety of other PC-related needs for staff throughout NIST. However, staffing levels in the Distributed Computing and Information Services Division appear to be on the low side to fulfill the division's agency-wide responsibility to provide such services to the desktops of more than 1,500 scientists, engineers, managers, and their related support staffs. Apparently, NIST has a two-tiered information technology support structure in place, where individuals in the various other operating units provide local assistance to supplement the support provided by the Distributed Computing and Information Services Division staff. The panel observes that this dual support strategy does not appear to be clearly articulated or widely understood. The appropriate balance between centrally supported services (the staff in this division) and departmentally supported services (individuals providing local help) is hard to achieve in any organization. However, it is important for NIST management to understand that reasonable support must be provided to achieve the effective use of a complex distributed computing environment. If enough support is not provided from the Distributed Computing and Information Services Division because of lack of resources, individual scientists will spend a significant amount of their time serving as their own computer support staff. Determining the appropriate long-term balance between central and distributed support should be an explicit goal of senior NIST managers. The panel continues to recommend that computer systems be upgraded every 3 to 4 years. Appropriate funding for upgrades needs to be allocated to the operating units, as a state-of-the-art organization should not be relying on 386 and 486 machines in 1999. By increasing the uniformity of hardware and software in use on the NIST campus, management will reduce the complexity of maintenance and improve the effectiveness of staff by limiting the number and types of systems and programs in which the Distributed Computing and Information Services Division must maintain expertise.
OCR for page 212
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Software Diagnostics and Conformance Testing Division Mission According to division documentation, the mission of the Software Diagnostics and Conformance Testing Division is to develop software testing tools and methods that improve quality, conformance to standards, and correctness; to participate with industry in the development of forward-looking standards; and to lead efforts for conformance testing, even at the early development stage of standards. The division's work on designing conformance tests is state of the art and clearly fulfills the division's mission and is consistent with the goals expressed in the laboratory and NIST missions. Technical Merit and Appropriateness of Work The Software Diagnostics and Conformance Testing Division has done an excellent job determining which standards merit attention. The division has emphasized newer technologies, and without exception, the programs can be expected to have widespread industry impact. A notable improvement since last year's assessment is the decision to focus Java-related efforts specifically on producing a requirements document for “Real-Time Java” and to move away from work on virtual machine interoperability, where there is already a great deal of industry activity. Research on language requirements for real-time systems and formal modeling contributes to the unique and valuable role NIST plays as a neutral party in Java language standardization efforts. The VRML Conformance project represented an innovative and effective way to present Internet conformance tests, semantic requirements, and standards. Now that automatic test generation tools have been completed, the project has reached an ideal stopping point. The expertise developed during the VRML work will be of significant benefit to the recently initiated XML effort. Another new project, Object-Oriented Technology for Distributed Interactive Learning, aims to define specifications that allow educational objects to be shared across the Web; the division hopes that this work will have a broad impact on education and advanced learning systems. Because software applications are embedded in many systems that affect safety and security, work that furthers software quality and reliability is imperative. Research in the division that advances software dependability includes statistical testing, formal methods, architecture description language (ADL) standardization, and failure data analysis. Formal methods work in the division is grounded in the search for practical techniques that will scale when applied to large, complex systems. Automatic generation of specification-based tests, using formal methods and a model checker, is an example of how NIST's expertise in this area can be used to benefit industry. In ADL work, advances could result in a great increase in software quality and reliability, since formal ADLs support simulation and model checking. Because the standardization of an ADL would strengthen component-based software development by the community at large, and thus provide significant productivity gains, research in this area could have a high payoff for the software industry.
OCR for page 213
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 A joint project with the Statistical Engineering Division is investigating whether statistical analysis can provide assurance that software that passes conformance tests is indeed meeting specifications. This work has won some support through a NIST Director's Competence Award, but significant results depend on the availability of sufficient funding. The momentum that was observed last year has been lost because of a reduction in the amount of staff time allotted to this project. Collection and analysis of reference data on software failures and provision of statistical methods and tools for analysis on software systems is an important contribution toward fulfilling society's dire need for higher-quality software. Difficulty in obtaining data on software failures is a problem for NIST and for forensic research in general. The panel recommends that NIST, in conjunction with industry and government, hold workshops in order to share failure data. This type of information is essential if the software industry is to learn from its mistakes. The panel is moderately concerned about the nondisclosure restrictions surrounding the software copymarking project. The project involves a very small expenditure, may have significant industry impact, and is being done with a company that has received an ATP grant. These factors appear to make this situation acceptable for ITL involvement. However, the panel would like assurance that nondisclosure work is generally discouraged and that this project is a carefully considered exception. Among other potential drawbacks, nondisclosure agreements may limit effective participation in broader colloquia. Impact of Programs Division projects are currently having a positive impact on industry in a variety of ways. A large number of companies report significant benefits from the division's Computer Graphics Metafile test suite. The RBAC model or its reference implementation has been adopted by several companies, including IBM, Sybase, and Siemens Nixdorf. The NIST document, Requirements for Real-time Extensions to the Java™ Platform,6 is being used by two industry splinter groups as the basis for their Java real-time implementations, which greatly increases the likelihood that the two implementations will be interoperable. NIST's early intervention in this process may prove to have been a vital step toward the ultimate widespread use of this technology. As part of the VRML effort, NIST formed and chaired the Conformance Working Group within the VRML Consortium. Microsoft, Netscape, SGI, Sony and other companies all are using NIST-developed conformance tests as a metric for testing VRML browsers. The VRML test suite receives 10,000 hits per month from 1,200 unique clients. Since this number greatly exceeds the number of organizations doing VRML product development, it is likely that the site is also used for other purposes, such as three-dimensional graphics education. 6 U.S. Department of Commerce, Technology Administration, National Institute of Standards and Technology, Requirements for Real-time Extensions for the Java Platform: Report from the Requirement Group for Real-time Extensions for the Java Platform, Draft NIST Special Publication (April 27, 1999), National Institute of Standards and Technology, Gaithersburg, Md., 1999.
OCR for page 214
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Resources Funding sources for the Software Diagnostics and Conformance Testing Division (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 4.6 4.7 Competence 0.5 0.6 ATP 0.4 0.1 OA/NFG/CRADA 2.2 2.0 Other Reimbursable 0.2 0.0 Total 7.9 7.4 As of January 1999, staffing for the Software Diagnostics and Conformance Testing Division included 39 full-time permanent positions, of which 34 were for technical professionals. There were also nine nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The panel was impressed with the high level of enthusiasm displayed by the staff of the division. Of course, noncompetitive salaries make it difficult to attract new computer science graduates to NIST. In the past, the division has circumvented this issue by hiring students as supplemental or part-time employees, who then are inclined to join NIST permanently after graduation. Statistical Engineering Division Mission According to division documentation, the mission of the Statistical Engineering Division is to advance scientific and industrial research by applying statistical methods to the collection and analysis of data critical to NIST scientists and collaborative partners in industry. The Statistical Engineering Division is involved in an impressively broad range of activities, involving collaborations with scientists throughout the NIST laboratories. Division staff continue to provide critical support in the Standard Reference Materials and Calibration Programs that are critical to NIST's mission. Over the past year, the division has made a concerted effort to increase collaborations with other divisions within ITL. Such projects now constitute roughly 15 percent of the division's work, still considerably below the long-term goal set by the laboratory. Nonetheless, the number of joint projects with other ITL divisions is growing, and NIST and ITL management have recognized that it is appropriate for more than half the division's effort to remain devoted to collaboration and consulting projects outside the ITL.
OCR for page 215
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Technical Merit and Appropriateness of Work The Statistical Engineering Division aims to maintain world-class expertise in statistical methods for measurement science and technology while identifying cutting-edge statistical techniques in relevant areas and demonstrating their utility. Many division programs provide clear evidence that the division continues to be successful in meeting these goals. The following six examples particularly impressed the panel: The work on Inference on a Common Mean in an Interlaboratory Study is ongoing, leading-edge methodological research in an area that is of significant importance to NIST. This project won the American Statistical Association's Youden Prize in 1998. A joint project with the Computer Security Division focuses on the development of a Statistical Test Suite for the Validation of Cryptographic Random Number Generators. This 2-year project involves the development both of new statistical methods and of a suite for testing existing algorithms. It appears likely that the results from this project will be incorporated into the ANSI X9.82 standard on random number generators. With personnel from the Software Diagnostics and Conformance Testing Division, Statistical Engineering Division staff are working on Software Testing Protocol Comparisons and Reliability of Conformance Tests. These projects are good examples of research aimed at comparing and evaluating state-of-the-art statistical techniques in the literature, demonstrating their utility in the context of NIST-related applications, and developing new methods as necessary to fill gaps in the literature. Unfortunately, the Statistical Engineering Division appears to have scaled down the amount of resources committed to this project, and the panel notes that this has adversely affected the progress of the project. In collaboration with the Optoelectronics Division of the NIST Electronics and Electrical Engineering Laboratory, the effort on Estimation of Time-Based Distortion and Alignment of Noisy Signals represents a novel methodological contribution to NIST measurement capabilities for the characterization of photodiodes. A 5-year competence grant from the NIST Director's office for work on Computational Metrology of Manufactured Parts was completed this year. This effort involved contributions from the Statistical Engineering and the Mathematical and Computational Sciences Divisions in ITL, as well as the Precision Engineering Division of the NIST Manufacturing Engineering Laboratory. This joint project has led to important advances in methodology and software for calibrating coordinate measuring machines and can be expected to have a significant impact on the manufacturing industry. Finally, NIST and SEMATECH (semiconductor manufacturing technology [consortium]) staff are developing and publishing the online Handbook of Engineering Statistics. This innovative project has great potential for use in training and educating scientists and engineers in industry to learn and properly utilize modern statistical methods. The handbook appears to be such a valuable teaching tool that the panel recommends that it be integrated into the Statistical Engineering Division' s in-house training for NIST scientists. A new project on Bayesian Metrology is too young for the panel to definitively assess its value. The division just received a 5-year competence grant to support this work, which is aimed at developing new Bayesian methods to address generic metrological problems facing NIST scientists. This project could potentially provide an improved and more sophisticated alternative
OCR for page 216
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 to the current, oversimplified procedures recommended in the ISO's 1993 Guide to the Expression of Uncertainty in Measurement. The panel observed that the Statistical Engineering Division is still struggling to find its appropriate niche within ITL. Although a “bottom-up” approach to project selection and management has worked well for the division in the past, there needs to be more careful, systematic planning if the division is to mesh seamlessly into the ITL of the future. Since there is a long list of traditional and newer demands for statistical services, priority setting must assume an increasingly important role in order for this division to achieve the highest possible impact and make the best use of its available human and fiscal resources. The panel believes that the division should develop a strategic plan that delineates future directions and includes clear guidelines on project selection and project management. As the Statistical Engineering Division supplements its traditional work on scientific statistical consulting with newer research focused on information technology, the panel offers a cautionary note about work in this highly popular and fast-moving area. The division would be wise to concentrate on opportunities where it has a special position of advantage or a unique set of expertise. NIST is neither a “software factory” nor a “technology manufacturer” and hence should beware of falling into the trap of undertaking research that could eventually be judged by industry to be irrelevant. Impact of Programs Through its extensive collaborations with other divisions and laboratories at NIST, the Statistical Engineering Division has a substantial and varied indirect impact on the scientific community at large and on industry in particular. In August 1997, a Web-based service that provides reference data sets for public use was initiated. This highly popular service receives approximately 11,000 hits per month and is likely being used by vendors of statistical software to improve the numerical accuracy of their products. Division scientists continue to publish at a reasonable pace in mainstream statistical journals as well as in journals focused specifically on applications. The innovative NIST/SEMATECH online Handbook of Engineering Statistics will have a large impact on the practice of statistics among scientists and engineers, and the panel urges the division to fully leverage this project into its educational efforts within NIST. The Statistical Engineering Division continues to provide extensive consulting support to NIST scientists for their more routine statistical investigations. Division staff conduct workshops aimed at training NIST scientists in basic statistical methods as well as workshops and conferences on more focused research topics. The director of the ITL has suggested that one technique for satisfying the long list of routine requests for statistical services is to have NIST scientists do more of the statistical work themselves, perhaps with the help of automated software packages supplied by the Statistical Engineering Division. This concept is appealing, but it is unlikely to be practical for any situations other than the most narrowly focused and repetitive tasks. However, the panel believes that targeted educational efforts, which could possibly include some automation or tailoring of software, may be a productive first step towards reducing the burden placed on the division by routine statistical requests. For example, the NIST/SEMATECH online Handbook of Engineering Statistics could be a valuable tool in such an educational effort. Regardless of the implementation details, the panel encourages NIST to continue to look into solutions for this problem.
OCR for page 217
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 Resources Funding sources for the Statistical Engineering Division (in millions of dollars) are as follows: Fiscal Year 1998 Fiscal Year 1999 (estimated) NIST-STRS, excluding Competence 2.8 2.8 Competence 0.3 0.5 STRS-Supercomputing 0.5 0.0 OA/NFG/CRADA 0.0 0.9 Total 3.6 4.2 As of January 1999, staffing for the Statistical Engineering Division included 23 full-time permanent positions, of which 21 were for technical professionals. There were also eight nonpermanent and supplemental personnel, such as postdoctoral research associates and part-time workers. The division staff includes five people who have joint appointments at NIST and as faculty at various universities. These personnel are making a significant contribution to the work of the division, and the panel strongly urges continued funding for this sort of arrangement. To increase its impact in the information technology arena, the Statistical Engineering Division will need to consider hiring staff with backgrounds in both computer science and statistics. The current personnel have limited capabilities of this kind, as is the case in virtually all academic and industrial statistics units. Furthermore, the demand for the small pool of people with this sort of expertise is extraordinary. Special effort and considerable salary flexibility will be required to hire staff in this area. However, if increased attention is paid to priority setting and the amount of more routine work is reduced, the division will be in a position to increase the focus on projects with novel statistical aspects, which, in turn, will improve morale and make the division more attractive to inventive recruits. One area of concern to the panel is the relative isolation of the Statistical Engineering Division in its current location at NIST North. Given the interdisciplinary nature of much of the division 's work and its extensive collaborations with other scientists at NIST, the panel strongly urges NIST management to consider relocating this division to the main campus. MAJOR OBSERVATIONS The panel presents the following major observations. The Information Technology Laboratory (ITL) and NIST management should explicitly recognize and support the varied roles of the divisions within the ITL. Performance measurement and resource allocation must take into account the differing work profiles within the divisions. The NIST director's visits to the staff to clarify the roles of individual divisions have been very helpful and should be continued as much as possible for as long as is necessary. In particular, the panel noted that the collaborative and research functions of
OCR for page 218
An Assessment of the National Institute of Standards and Technology Measurement and Standards Laboratories: Fiscal Year 1999 the Statistical Engineering Division still need to be more fully integrated into the ITL's activities and mission. NIST's historic culture of openness creates a potentially serious problem when applied to its computer networks. The current firewall situation is inadequate and leaves NIST open to a potentially damaging and embarrassing intrusion. The process of ensuring that the entire organization is protected by a centrally maintained firewall system is proceeding much too slowly; strong managerial action is required to remedy this dangerous situation in an efficient and timely manner. The ITL continues to refine the process by which older programs are selected for termination and new areas are chosen for funding. The panel applauds the progress that has been made so far and supports efforts to continue to improve the process. Several divisions in ITL have experienced difficulties in interacting with the Department of Commerce's legal services divisions. The two examples in most desperate need of being resolved soon are the year-long failure to finalize the arrangements with the University of Maryland for a joint institute for information technology and the 2-year delay in approval for NIST to join the World Wide Web Consortium. Other cases of inefficiency, such as months taken to obtain a nondisclosure agreement, have also been observed. The panel again recommends that the ITL more actively gather data on the results and consequences of its technical activities. More organized, quantitative tracking of interactions and outcomes would make it easier for NIST to justify the value of the work performed in this laboratory. Information technology is a dynamic area with an extremely high demand for skilled people in all roles. NIST must find creative ways to attract and retain employees at all education and skill levels; this effort should include exploring ways to arrange increased compensation for such employees. ITL management has been especially responsive to the panel's previous suggestions and to its requests for information. The panel is grateful for this excellent cooperation.
Representative terms from entire chapter: