Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 257
1 ~ Information Technology Laboratory Division Reviews . MATHEMATICAL AND COMPUTATIONAL SCIENCES DIVISION Technical Merit The goal of the Mathematical and Computational Sciences Division (MCSD) is to provide technical leadership in analytical and computational methods to solve scientific and engineering problems of interest to NIST and to U.S. industry. The technical strength of the division's staff, the breadth of its portfolio of projects, and the technical merit of its individual projects are impressive. These projects include collaborations of varying duration with other NIST scientists, the development of tools and standards in mathematical software and computation, and an imaginative exploration of architectures and algorithms for quantum computation. The division organizes its work around five technical programs: three general areas (Applied Math- ematics, Mathematical Software, and High-Performance Computing and Visualization) and two spe- cific, large projects (Quantum Information and the Digital Library of Mathematical Functions). The four groups of the division Mathematical Modeling, Mathematical Software, Optimization and Computa- tional Geometry, and Scientific Applications and Visualization correlate roughly with these technical programs, but many projects cross groups, divisions of the Information Technology Laboratory, and NIST laboratories. In the previous two review cycles, this panel recommended that the division strengthen its strategic planning, especially since the demand for the division's work outstrips its resources. The division's response to this recommendation has been outstanding: the draft of its triennial strategic plan shows careful attention to the selection of projects and their life cycles and considers internal and external NOTE: Chapter 7, Information Technology Laboratory," which presents the laboratory-level review, includes a chart showing the laboratory's organizational structure (Figure 8.1) and a table indicating its sources of funding (Table 8.1~. 257
OCR for page 258
258 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 customer needs as well as relevant trends and context in applied mathematics. Division staff at all levels participated in developing the plan, which appears to be guiding the division's activities. Many of the projects in the Applied Mathematics Program are collaborations between MCSD mathematicians and domain scientists in other parts of NIST. For example, the long-running, collabora- tive work with MSEL on solidification modeling has yielded significant advances in modeling capabil- ity and is recognized as excellent in both the applied mathematics and materials science communities. The Object Oriented Micromagnetic Framework (OOMMF) tool for micromagnetic modeling has had extensive impact both on NIST collaborations with MSEL and KEEL and on external customers. A long-running collaboration with BFRL on construction metrology is achieving good results. Much of the Mathematical Software Program is currently focused on the Digital Library of Math- ematical Functions (DLMF), an ambitious project to produce an online (Internet) replacement for the classic and still-much-used Abramowitz and Stegun Handbook of Special Functions. This important project is nearing fruition; the online handbook's release is scheduled for 2004. Successfully managing this extensive project is a significant accomplishment. DLMF continues MCSD's tradition of innovative use of the Internet to support the applied mathematics community. This tradition includes the Guide to Available Mathematical Software, which remains one of the most heavily used NIST Web sites; the Matrix Market, which disseminates standard test data for numerical linear algebra; and the newer Template Numerical Toolkit. The Mathematical Software Program has also been very active in standards activities, such as the Java Numerics effort. This remains an area in which NIST is able to have a positive effect on a fast- moving commercial standard primarily because of the strong reputation of division scientists in the technical community and because of the Sparse Basic Linear Algebra Subprograms standard, approved in 2003. The Mathematical Software Group has lost personnel in the past few years; the panel hopes that the group will be able to sustain its high level of contribution beyond completion of the DLMF. The Scientific Applications and Visualization Group joined MCSD about 2 years ago. This group provides a crucial source of expertise for NIST scientists in a wide range of disciplines, from fundamen- tal physics through the modeling of cement and concrete. The latter work includes several projects in collaboration with BFRL and involves a sizable consortium of industrial partners. The reorganization of this group into MCSD seems to be going very well from all points of view and to be viewed positively by the staff involved. The group's opportunities for interaction within the division are, however, made difficult by its physical separation from the rest of MCSD, which is located in NIST North, away from the main NIST campus in Gaithersburg. The Quantum Information Program, performed in collaboration with NIST physicists and other ITL divisions, is very forward looking research that is necessarily somewhat speculative but has enormous potential importance; NIST is ideally situated to conduct this effort. The quantum bus work in MCSD is a good example: whether or not the specific approach under investigation is successful, this work represents a valuable attempt to identify and grapple with the architectural issues in quantum comput- ing. MCSD has completed a good draft of a strategic plan and has introduced a new rating system to grade whether projects are in active development. The division has addressed the need to track the life cycle of projects by creating ratings for projects indicating their maintenance levels. Program Relevance and Effectiveness The Mathematical and Computational Sciences Division continues to provide high-quality service for its customers, which include other NIST laboratories, industry and academic partners, and the
OCR for page 259
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 259 scientific community. Almost all of the projects in the division are highly relevant to customers in one of these categories. Examples include the work on solidification for the Materials Science and Engineer- ing Laboratory, visualization of concrete hardening for the Building and Fire Research Laboratory, OOMMF application to a high-profile physics project at the University of Durham, and support of the thousands of scientists who use mathematical resources on the division's Web site. The division's Quantum Information Program has considerable scientific relevance. In principle, quantum computers can solve problems much faster than can traditional computers, because the regis- ters of the former can be in multiple superposed states at the same time. Several different physics laboratories have tried to build realizations of quantum computers, with varying degrees of success. The division's research in this area is relevant to this exciting line of scientific inquiry. Measures of the relevance of MCSD work also include awards received and publications contrib- uted by the division staff during the past year. These include an award for best paper for work in construction management techniques and a NIST Bronze Medal for the OOMMF project. The number of published papers has grown from 12 (last year) to 16. Although the majority of MCSD projects reviewed seemed to have impressive relevance and effec- tiveness, the panel was not able to determine whether this is the case across all projects. Therefore, the panel proposes that descriptions of projects should include a discussion of their relevance to customers, to the scientific community, or to high-profile scientific issues. Division Resources Resources for the MCSD continue to be flat. The panel was not able to determine whether there is shrinkage, because the delay in passing the 2003 congressional budget is still having a ripple effect on all divisions. There is a personnel shortage in the division, particularly in the Optimization and Compu- tational Geometry and the Mathematical Software Groups. The position filled by the acting leader of the Mathematical Software Group should be filled by a permanently appointed leader. The situation in the Optimization and Computational Geometry Group is severe because previous personnel losses have not been compensated for. One indication of the problems for this group is that its leader is the division chief, and no plan for changing this arrangement has been presented to the panel. Although there is enough funding to complete the DLMF, funding for its long-term maintenance as well as that for other division Web sites is of concern. Within externally imposed constraints on personnel, the division chief is doing an excellent job of allocating scarce resources. In particular, the panel sees the hiring of postdoctoral researchers in quan- tum computing and external authors for the DLMF as good ways to draw benefit from a barely adequate budget. Despite the constraints, morale appears to be high, seeming to reflect focused efforts by the . . . . .~ c .lvlslon c. clef. MCSD staff members are dissatisfied with the "e-approval" system widely used at NIST. The system apparently has not had the desired effect of facilitating paperwork tasks, and staff consider its Windows operating system incompatible with the Linux used by many division researchers. Staff concerns about the split of division personnel between NIST and NIST North, as well as the larger split between Gaithersburg and Boulder, appear to have diminished. The panel continues to recommend that NIST management relocate MCSD to the main NIST campus in order to increase the opportunity for building the cross-laboratories collaborations that are its lifeblood.
OCR for page 260
260 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 ADVANCED NETWORKING TECHNOLOGIES DIVISION Technical Merit The goal of the Advanced Networking Technologies Division is to provide the communication networking industry with test and measurement research and technology. The division consists of three groups: High Speed Network Technologies, Wireless Communications Technologies, and Internetwork Technologies. The division's work is currently organized in six projects: Networking for Pervasive Computing, Wireless Ad Hoc Networks, Agile Switching Infrastructures, Internet Infrastructure Protec- tion, Internet Telephony, and Quantum Information Networks. The division focuses on using and developing test and measurement tools, technologies, methods, and metrics to guide and improve the quality of networking specifications and standards and to improve the quality and interoperability of commercial networking products. It also models, simulates, proto- types, and conducts empirical studies to help ensure that specifications for emerging technologies produced by industry and standardization organizations are complete, unambiguous, and precise. The division' s work encompasses several of the currently important areas in networking research and stan- dardization. The work of the Advanced Networking Technologies Division is of consistently high quality and exhibits continuing improvements. The organization of ongoing research around coherent research themes has produced good synergy, fostered communication and collaboration among the research groups, and provided continuity as projects are completed and new activities are initiated. The projects are well focused on achieving specific goals. Two activities form the core of the division's work on Networking for Pervasive Computing; both activities support the development of networking standards. The first focuses on crafting wireless standards, including IEEE 802.15 Wireless Personal Area Networks and IEEE 802.11 Wireless Local Area Networks, so that they do not conflict within the unlicensed 2.4-GHz radio-frequency band. NIST has taken a leadership role in reconciling relevant standards, and division staff have contributed to a document that recommends practices for designers of relevant standards-compliant communications equipment. Division staff reported that there is increasing industry adoption of designs exploiting the NIST recommendations for synchronized receivers and combined radios. The project has begun to model the performance of dynamic traffic loads at the network layer and has initiated work on ultra- wideband (UWB) communications systems and protocols, exploring interoperability and overlooked interference issues between UWB and existing narrowband communications systems. Division staff are participating in the IEEE 802.15.3a study group on network coexistence. Early entry into UWB coexist- ence studies is appropriate and well matched to the division's competencies. The timely availability of new technical information will allow the IEEE groups to incorporate the division's solutions into the standards. The second activity in the area of Networking for Pervasive Computing focuses on the analysis of the resource discovery protocols being developed for ubiquitous computing systems. Division staff are analyzing and simulating protocols to evaluate their functions and how they scale with different network sizes. Current activities include work on modeling the robustness of Jini™ and Universal Plug-and- Play™ in the presence of node failures and developing a simulation model for the Internet Engineering Task Force's standard Service Location Protocol. Division staff have analyzed the performance of service leasing to better understand the trade-off between protocol scalability and service guarantees. The division has responded to the panel's suggestion to extend its investigation beyond proprietary service discovery protocols to include the Service Location Protocol. The Networking for Pervasive -r ------- - - --- - - ------I
OCR for page 261
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 26 Computing project hosted an industry workshop on pervasive computing for a second consecutive year. Overall, this project is of continuing high quality. The Wireless Ad Hoc Networks project develops technologies and standards for wireless ad hoc networks (WANETs) and explores video communications in sensor networks. Research activities in ad hoc networks have focused on the development of clustering algorithms and structures and on routing techniques in multihop settings. Division staff members actively participate in the IETF Mobile Ad Hoc Networks working group and have provided valuable results for the ad hoc on-demand distance vector (AODV) routing algorithm to that group and other standards groups. The division is also codeveloping a lightweight AODVjr protocol with external collaborators in the research community. The project team has also continued using and enhancing WANET simulation environments and has contributed to an improved model for IEEE 802.1 1 local area networks in the popular, commercially available OPNET™ network simulation tool. Research activity in the sensors area has focused on improving the quality of video communications and on developing protocols for self-organizing sensor networks. The Agile Switching Infrastructures project conducts performance modeling and evaluation of optical networking technologies, focusing on network control, configuration, and management. The division continues to develop and enhance two separate software research tools. The first tool, NIST Switch, provides an emulation platform for multiprotocol label switching (MPLS) optical networks in support of traffic engineering studies. Division staff have successfully used this tool to simulate service quality differentiation between traffic classes transmitted over MPLS networks. Division staff also support and update the GLASS (GMPLS Lightwave Agile Switching Simulator) optical network simu- lation tool, which promises to be a key research tool for analyzing approaches to network restoration and recovery. In an effort to promote the use of GLASS in the external research community, this past year the project team hosted a workshop for GLASS users. The panel will look for additional evidence of wider external adoption of GLASS in the coming year. The Agile Network project has also initiated a timely and promising activity on the evaluation of optical burst switching techniques. The Internet Infrastructure Protection project continues its collaborative work with ITL' s Computer Security Division on critical network infrastructure protection, including the protection of the Domain Name System with DNS Security (DNSSec). An ongoing activity to evaluate the performance and scalability of the Internet Protocol Security (IPSec) key management protocols has resulted in the creation of a simulation environment for IPsec and Internet Key Exchange (IKE) Versions 1 and 2; development of this simulation framework is now complete. The project's staff participates in relevant IETF working groups; this activity can help IETF participants select a successor to the IKE key manage- ment protocol. The staff has made good decisions to investigate secure routing protocols and to shift emphasis from modeling operation of the existing protocols to investigating how protocols will respond to attacks. As part of the NIST-wide initiative in quantum computing, the Quantum Information Networks project is collaborating with the Computer Security Division on protocols and prototypes for quantum cryptography. The division's principal contribution is in the area of key management protocols for quantum key distribution and focuses on an improved communication channel model and the proper selection of error-correcting codes for that channel. Division staff have devised an alternative reconcili- ation algorithm to the state-of-the-art Cascade algorithm. The division has also supported the NIST testbed operation by maintaining and upgrading the wavelength division multiplexed transmission equipment and testing the free-space optical transmission link. This project has both a protocol design and a prototyping element using a real quantum channel. Although the practical impact of this work is too far in the future to predict, having a promising, long-term project that complements short-term
OCR for page 262
262 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 projects gives good balance to the overall research program. Division staff should remain alert to potential synergies between the Quantum Information Networks project and other projects. The Internet Telephony project has again made considerable progress with its continuing focus on call signaling protocols. The Session Initiation Protocol (SIP) interoperability test tool is used and valued by the voice-on-If industrial research community, and the Web-enabled SIP load generation and trace capture elements of this tool are demonstrating their utility in helping implementers debug subtle interoperability problems. The examination of network service programmability through SIP-JAIN (JAVA Advanced Intelligent Network) and the investigation of dynamically scripted SIP agent tech- nology represent a new direction in SIP work. Work also progressed during the past year on the NIST SIP-lite prototype for limited capability devices such as handheld computers. Current research toward XML-based call flow descriptions will help the project expand beyond Java-specific programming environments. The Internet Telephony project continues to be a model for collaboration with industry. The project is also successfully balancing the maintenance of existing software tools with the need to advance its research agenda. In the coming year, the division has proposed to introduce exploratory projects on first-responder networks, network metrology and measurement, and networking for grid computing. These ambitious initiatives represent promising opportunities for collaborative work with other NIST laboratories, with an associated challenge of developing effective mixes of short- and long-range goals. Program Relevance and Effectiveness The staff of the Advanced Networking Technologies Division continues to be active in industry organizations, including the IETF, IEEE, and the International Telecommunication Union. Division personnel are well respected by the staff of these standards organizations and by the communities they serve. The value of the division's standards-related efforts is realized in several ways. Most often, technical work done at NIST, such as modeling and analysis or development of testing tools and evaluation criteria, provides a greater understanding of the implications of proposed standards or sup- plies solutions to problems that could arise in standards development. The division's familiarity with the networking community and its reputation for an unbiased technical approach are also useful in deter- mining what issues have inspired the standards effort and in defining the technical matters on which the standards bodies should focus. A recent example of such impact is the division's leadership within the IETF's investigations of DNSSec and IPSec, cumulatively leading to the publication of seven IETF requests for comments. In its previous assessment, the panel continued to discuss industry's practice of developing stan- dards in consortia or other private groupings rather than through the traditional "open" approach that primarily involves professional organizations. There is recent indication that consortia are addressing participation issues affecting NIST, with the Java Specification Participation Agreement cited as a model for other consortia or groupings. The division recognizes that the "closed" system is somewhat antithetical to the NIST and to the governmental philosophy of supporting all U.S. companies and the public in an open manner. However, to support the NIST mission of strengthening the U.S. economy, the division must be able to influence the standards used by the networking community. NIST should continue to communicate its position on this matter to relevant consortia and to define criteria for participation in these consortia. The Advanced Networking Technologies Division assumes a leadership role in the networking community in part because of its standards-related activities. However, it is important for the staff to expand and maintain its reputation in other ways as well. During the past year, the division staff
OCR for page 263
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 263 increased its publications in journals and conference proceedings and attended relevant meetings. These activities are appropriate and responsive to the panel's previous suggestion that the division strengthen its presence in the most prestigious publications and conferences of the networking field. The division should maintain this increasing level of visibility in the community. Division researchers have also participated in professional service activities as technical program committee members, workshop organizers, and journal editors. Division Resources As of January 2003, staffing for the Advanced Networking Technologies Division included 25 full- time permanent positions, of which 21 were technical professionals. There were also 10 nonpermanent or supplemental personnel, such as postdoctoral research associates and temporary or part-time workers. Two new full-time permanent researchers joined the division staff. Approximately 53 percent of the division's research staff hold doctoral degrees. The contributions of the division are limited partly by the fact that the number of full-time perma- nent staff is modest. The division's relevant and effective work is due in part to a large cadre of guest researchers (23 people as of February 2003~. This heavy reliance on visitors means that the division depends on temporary employees to support mission-critical projects, and there is the potential for unexpected delays or the premature termination of an important effort when a guest researcher leaves NIST. These risks are currently outweighed by the benefits provided by the added personnel and the relationships built with other institutions, but the division should continue to be careful about maintain- ing an appropriate balance between permanent and temporary staff. Division management has noted the absence of a program to hire U.S. citizens as guest researchers at a time when the troubled economy has increased the pool of potential hires. It has also observed that the hiring process for foreign students remains considerably easier than that for U.S. students, and this policy decreases NIST's ability to maintain the closest possible ties to U.S. universities. NIST should revisit its current hiring policies and programs to continue to ensure that it achieves the highest possible level of technical excellence in new hires. Division management expressed some concern over both current and future year budget uncertainties that make it difficult to plan staffing targets. Management also reported that the narrowing spending guidelines associated with external agency funding are further limiting its ability to use those resources as efficiently as in the past. Morale within the division appears to be good; the staff is enthusiastic about its work and proud of its accomplishments. Prior concerns about the adequacy of the internal information technology infra- structure did not arise. The position of leader for the High Speed Network Technologies Group has been filled, which facilitates the group's stability and focus. COMPUTER SECURITY DIVISION Technical Merit The goal of the Computer Security Division is to improve information systems security by raising awareness of information technology risks, vulnerabilities, and protection requirements; advising agen- cies of information technology vulnerabilities; devising techniques for the cost-effective security and privacy of sensitive federal systems; developing standards, metrics, tests, and validation programs to promote, measure, and validate security in systems and services; and developing and disseminating guidance for managing secure information technology. The division's programs directly support these
OCR for page 264
264 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 goals. The recent emphasis on homeland security has given more impetus and visibility to the division's work. The programs under way in the Computer Security Division are appropriate and of high technical merit. The division is composed of four groups: Security Technology, Systems and Network Security, Security Management and Guidance, and Security Testing and Metrics. The Security Technology Group's projects fall into the areas of cryptographic standards and guide- lines and of public key infrastructure and applications. Its cryptographic toolkit provides algorithms and techniques to U.S. government agencies and others. The group's modes and key management guidelines (such as SP 800-38B, Recommendation for Block Cipher Modes of Operation) are essential to ensure secure, interoperable crypto implementations in products that are evaluated under the division's Crypto- graphic Module Validation Program (CMVP) and its National Information Assurance Partnership (NIAP). Work is ongoing in the area of key establishment and management, where FIPS (Federal Information Processing Standards) 140-2 coverage is needed. There is an ongoing discussion about whether work to define block cipher modes should focus on T-DES (the Triple Data Encryption Stan- dard) and AES (the Advanced Encryption Standard) separately or attempt to satisfy the needs of both algorithms with a single mechanism. The progress on AES is critical. NIST should design separate mechanisms, working on T-DES only where there is a clear and ongoing commercial requirement. The Security Technology Group's e-authentication effort is providing technical guidance on this standard component in government enterprise architecture in the areas of protocols and credentials. This effort has to take into account the September 2003 report of the NRC's Computer Science and Telecom- munications Board's Committee on Authentication Technologies and Their Privacy Implications, Who Goes There? Authentication Through the Lens of Privacy (National Academies Press, Washington, D.C.) The threat model for the e-authentication effort needs to be clearly and explicitly articulated. The recommendations arising from this effort need to take into account the consequences not only of false positives (accepting an inauthentic individual as authentic) but also of false negatives (rejecting an authentic individual). The Systems and Network Security Group is working in a broad range of areas, including emerging technologies, reference data and implementations, and security guidance. The group's work on system _ ~ _ _ ~ ~ . . . .. . ~ ~ - ~ a. ~ ~ ~~ ~ ~ . ~ . ~ ~ . ~ ^. .. .. ~ ~ aclmmlstratlon guidance tor windows zuuu ~rotesslonal estanllsnes a configuration with known secu- rity properties that can be used by system administrators, application developers, chief information officers (CIOs), and auditors. This work can influence Microsoft's secure configuration development efforts. The panel supports and commends this effort and the division's cooperation with Microsoft. The group's IPsec online test facility, developed jointly with the Advanced Networking Technolo- gies Division, is helpful to vendors. It is also a model for the future testing of protocols in that it imposes minimal ongoing costs to NIST and yields good benefits for industry. The group's work on reference data and intrusion detection might be directed toward providing a reference data set for testing intrusion detection systems (IDSs). NIST has often provided reference standards to industry for the calibration of instruments and, in the computer and communications arena, for standards conformance testing. In the field of information security, considerable effort is focused on technology for intrusion detection, yet there are no standard measures for the effectiveness of IDSs. The division should consider the develop- ment of a reference data set to support such evaluation for network-based IDSs. The work of the Security Management and Guidance Group is appropriate and of high technical merit. A primary focus of this work is the Computer Security Resource Center, a valuable Web site that provides information about computer security for the public. The group's NIST SP800-4 (Computer Security Considerations in Federal Procurement) gives government agencies guidance on how to incor- porate common-criteria security evaluation language into requests for proposals. The division is work-
OCR for page 265
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 265 ing on new language for the Federal Acquisition Regulations that purchasing officers can use verbatim. This is very important and challenging work. In 2002, the Security Management and Guidance Group published NIST SP 800-30 (Risk Manage- ment Guide for Information Technology Systems), which establishes good practice for industry and government and provides guidance to assist federal agencies in performing risk analyses as part of their security procurement and deployment planning. One of the key programs in the Security Testing and Metrics Group is the CMVP. This program has a quantitative focus and aims to provide automatic measures of compliance. It is an important effort that continues to uncover and correct a large number of flaws in algorithm implementation and documenta- tion. The process certifies cryptographic modules, providing a common assurance definition for custom- ers of those modules. The group validated more than 120 cryptographic modules and more than 150 cryptographic algorithm implementations in FY 2002. This is a high-growth area in which the division resources are stretched to their maximum capacity. Ongoing work on mutual recognition in CMVP certification has produced seven NIST-accredited CMVP laboratories: four in the United States, two in Canada, and one in the United Kingdom. Accredi- tation of laboratories outside the United States is important for the economic viability of products undergoing PIPS 140-2 evaluation. The division is also taking FIPS 140-2 to ISO. This will lead to international recognition of a U.S. standard, making companies that have already applied for and received the certification more competitive worldwide. Having a single standard makes the industry more efficient and improves the quality of cryptographic technology in critical infrastructure deploy- ments worldwide. The National Information Assurance Partnership Program is another important component of the Security Testing and Metrics Group. It focuses on developing common-criteria protection profiles and investigating issues related to the use of these profiles in developing security requirements for the federal government. Fifteen nations have signed mutual testing agreements that recognize the common criteria and the common criteria testing laboratories. The CMVP and NIAP work is relevant to the attempts to provide system architects with better building blocks for constructing secure systems. The division has adopted the panel's suggestion that CMVP and NIAP not be merged. Program Relevance and Effectiveness Privacy and security are essential to protecting electronic commerce, critical infrastructure, personal privacy, and private and public assets, and so the work of the Computer Security Division makes important contributions to strengthening the U.S. economy and promoting the public welfare. The division' s activities are relevant to a broad audience, including hardware and software makers and users in industry, the federal government, academic and industrial researchers, and the public. The division develops standards and guidelines for cryptography and security implementations, produces tools and metrics for testing compliance and performance of security systems and products, and supports the development of new and more effective security techniques. Division staff members disseminate their results through publications, presentations, advice to government agencies, participation on committees, and Internet posting of tools, databases, and information. The publication record of this group is excel- lent it estimates having published approximately 1,400 pages of guidelines within the past 2 years. Web site statistics suggest how effectively the division disseminates its results. Each month from January 2002 through February 2003, an average of 764,682 pages were returned and 1,281,396 re- quests were handled. Data from the Computer Security Resource Center Web site is accessed by federal agencies, businesses, and schools.
OCR for page 266
266 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 The Cryptographic Module Validation Program has i graphic products. About 50 percent of the cryptographic modules tested had security flaws, and over 95 percent had documentation errors. About 25 percent of the algorithms submitted for evaluation had security flaws, and over 65 percent had documentation errors. Detecting these problems enables vendors and implementers to correct their products before the modules and algorithms are put into production and bought and used by consumers. This program offers a sterling example of the division's relevance and effectiveness. Division staff are active in several national and international standards activities and in groups such as ANSI, ISO, and IETF. The committees and activities of these organizations are examples of open standards development and adoption environments. In the case of FIPS 140-2, taking it to ISO will lead to the international recognition of a U.S. standard that makes the industry more efficient. In other cases, it may be most appropriate for NIST to issue standards under its own banner. The division has been relying on the ANSI X9F Committee for the development of random-number-generation standards for key management. However, the X9F Committee is a financial industry committee whose standards might not always be generic and therefore may not always be suitable for promulgation as NIST guidelines. Division Resources improved the security and quality of crypto- As of January 2003, staffing for the Computer Security Division included 48 full-time permanent positions and 6 part-time permanent positions. In 2002, $4.7 million of the $15 million division budget was derived from external sources of funding. That ratio (31 percent) indicates a high level of relevance of division programs to its customers. The division expects that as funding from other sources is identified, 2003 funding will be approxi- mately the same as that for 2002. In 2002 the division chief implemented a successful CMVP cost- recovery program. Charging customers for cryptographic module validation has enabled growth in the number of CMVP certificates issued and improvement in the level of evaluation. This type of effort is needed to ensure that testing for important areas of security continues to be available despite chances in funding from other sources. O There is insufficient funding for pursuing certain important programs. The Cyber Security Research and Development Act of November 2002 created more responsibilities and requirements for the divi- sion, but little funding has been appropriated to accompany these authorizations. In addition, many of the programs at ITL (e.g., biometrics, health care, and voting system standards) have major security components. There is risk that the skills in the division may not be effectively leveraged in cross- division initiatives. For example, existing biometrics projects should address security and assurance considerations in collaboration with the Computer Security Division. The division chief noted that funding for research into security composability under the Cyber Security Research and Development Act has not been reauthorized for FY 2004. The division has ideas but no funding for improvements to the increasingly important NIAP evaluation and certification processes. An additional person is required by the Security Technology Group to conduct work in cryptography and by the Security Testing and Metrics Group to conduct CMVP validations. Many members of the division staff seem to appreciate that the quality of the facilities in NIST North exceeds that of NIST's main Gaithersburg campus. However, the distance from the main campus introduces difficulties for employees looking for other job opportunities within NIST. Staff have noted that opportunities for change are enabled by the kinds of informal interactions that occur within a facility, particularly for junior members of the team. This situation may be a contributor to the dissatis-
OCR for page 267
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 267 faction reported on the employee survey in the areas of opportunities for advancing at NIST, changing career paths at NIST, and fair administration of the job-posting process. In skip-level discussions, staff mentioned that helpful technologies such as instant messaging are precluded by the NIST security configuration; this should be fixed if feasible. O 0 1 INFORMATION ACCESS DIVISION Technical Merit The goal of the Information Access Division (IAD) is to accelerate the development of technologies that allow intuitive and efficient access, manipulation, and exchange of complex information by facili- tating the creation of measurement methods and standards. IAD is organized in four groups: Speech, Retrieval, Image, and Visualization and Usability. The Speech Group is conducting projects in the areas of Rich Transcription, Automatic Meeting Transcription, and Speech and Language Recognition. The Rich Transcription project is designed to support a 5-year research program, Effective, Affordable, Reusable Speech-to-text the EARS pro- gram sponsored by DARPA. This project is directed toward moving the state of the art from word recognition toward natural-language understanding for large vocabularies. An initial set of benchmarks and evaluations, RT-02, was run in 2002; it required competing systems to extract words from English speech provided by the Linguistic Data Consortium. This speech generally involved several speakers; the metadata extraction task was to partition the speech according to the speaker. Under the self-initiated Automatic Meeting Transcription project, a meeting room has been fitted with cameras, microphones worn by participants, desk microphones, and microphone arrays mounted near the walls, to provide data on the same meeting from several perspectives. The speech material collected will be turned over to the Linguistic Data Consortium, will form part of the 2003 testbed for EARS, and will be used in the speaker-recognition trials. The Speech Group is performing several related speech and language recognition tasks involving the processing of audio data or extended data (audio plus imperfect transcription of speech and other derived information). The goal is to improve the quality of software for recognizing the presence of a particular speaker or for determining the speaker's language. Much of this work is supported by the National Security Agency, and annual comparison tests have been held since 1996. In 2002, evaluations were held in which systems were asked to determine whether speech belonged to a given speaker, or whether a given speaker was one of two on a recorded conversation. Comparisons were made for audio- only and extended data; large improvements were seen when the extended data were used. However, gains over the previous year's results were reported to be small, suggesting that new approaches to the technology may be necessary. The goal of the Retrieval Group's Text Retrieval project is to evaluate and encourage research and technology transfer in new information access technologies. The best-known effort in this area is the Text Retrieval Conference (TREC), which is now in its eleventh year. TREC participation continues to grow; this annual workshop attracts a broad range of government, industrial, and academic participants. TREC provides a forum for leading researchers to evaluate systems by focusing on key common problems. It continues to be a hub for new research, and it helps focus researchers on government challenges (e.g., novelty detection, cross-language retrieval, and question answering). NIST is in a position to serve as an impartial facilitator and evaluator of research in this area. The continual evolution of the program, addressing challenges and pruning topics with diminishing returns, has contributed to its success. In 2002, TREC introduced two new tasks, novelty detection and genomics. The focus on
OCR for page 268
268 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 developing solid and reusable test collections and the push to attack new information access problems are the two important contributions of the TREC program. The Automatic Content Extraction (ACE), Advanced Question and Answering for Intelligence (AQUAINT), Document Understanding Conference (DUC), and Topic Detection and Tracking (TDT) efforts do not have the visibility of TREC because they are smaller, and participation is linked to funding. ACE aims to advance the state of the art in extracting content from newswires, broadcast news, and newspapers. In AQUAINT, the focus is on moving question answering to the full range of complex questions asked by analysts, not just the short, fact-based questions considered in TREC. In DUC, new corpora and evaluation methods are being developed by NIST to evaluate summarization. In TDT, NIST hosted the annual Topic Detection and Tracking Conference and analyzed and presented the evaluation results. Metrics to evaluate new technologies are critical to the success of these government-sponsored programs. The Retrieval Group is developing metrics and collections in challenging new areas such as extraction, summarization, question answering, and interactive systems. The Image Group's long-standing work and expertise in biometrics are now being sought to address requirements of the USA PATRIOT Act of 2001. This group is running operational and recognition rate tests on very large databases of fingerprint and face images. Test results are eagerly anticipated, because no test of this magnitude and with third-party expertise has ever been performed. Results will be used to make decisions regarding government use of biometrics for visas, border security, and future homeland security tasks. The biometric testing results will also affect a related industry, since a burgeoning group of companies hopes to capitalize on government security contracts. A related biometric activity is that of promoting an interoperability exchange format for fingerprints and palm prints. This work will affect any use of fingerprints or palm prints for public security, because different systems will be used by different adopters. The project on Human Identification at a Distance, which began before the current surge in interest in biometric surveillance, is an example of NIST's anticipation of future important areas. The Image Group' s multimedia work focuses on Motion Pictures Expert Group (MPEG) standards. The project's two staff members promote MPEG-7, chair two related committees, sponsor an MPEG workshop, are designing an interoperability testbed, and promote multimedia standards. The group's Pervasive Computing project has delivered two major items this year: a second version of the Smart Flow system (software complete with an application programming interface, so that anyone can quickly begin researching pervasive computing), and 20 hours of meeting data (at 70 GB per hour) for the Automatic Meeting Transcription project. The Visualization and Usability Group provides metrics, standards, and test methodologies to improve the usability of interactive systems. A shift in focus is under way from developing visualization techniques and usability testing to evaluation methodologies and standards to support usability and accessibility. This shift in focus is in good alignment with division's strengths, and the group is making good progress in the new direction. Usability is a high-impact area, because poor usability contributes to the high cost of ownership of software and to lowered efficiency. The Industry Usability Reporting (IUSR) effort and the Common Industry Format (CIF) for reporting summative user test results provide the infrastructure for sharing usability information between consumers and producers of software. The CIF standard lays the foundation for factoring usability into software procurement decisions. The Visualization and Usability Group was instrumental in bringing together industry leaders in a series of workshops and in driving the effort at standardization. ANSI approval has been achieved (ANSI/ INCITS 354), extensions are being considered, and ISO fast-track procedures are under way. To create benchmark test data for Web usability evaluation methods, the group analyzed the results of pilot studies on CIF testing, evaluation, and reporting. The group is also beginning to explore a framework and ~ ~ ~ 1 ~7
OCR for page 269
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 269 methods for evaluating complex interactive systems. Initial explorations in the area of intelligence analysis (AQUAINT and NIMD ENovel Intelligence from Massive Data]) and robotic interaction are focused on important government problems and are receiving financial support from the Advanced Research and Development Activity (ARDA), DARPA, and ARL. Program Relevance and Effectiveness DARPA is the sponsor for the rich transcription (EARS) program, and the National Security Agency sponsors the speaker recognition work. The Automatic Meeting Transcription project provides data for each of these, and will provide its data to the Linguistic Data Consortium. There are 17 universities and nonprofit research laboratories involved in the EARS program. The competitions have been open to others, and several U.S. and foreign corporations and schools have participated. The Speaker Recogni- tion project supports and evaluates the work of about 25 organizations, including corporations and several domestic and foreign universities. The Retrieval Group has a wide range of customers. The intelligence community's ARDA and government agencies such as DARPA work closely with the group to evaluate the success of new information access technologies funded by their programs. In addition, hundreds of participants from government agencies, industry, and academia take part in the annual TREC program. Participation in TREC continues to increase, and it evolves as new tracks are added and old ones are phased out to reflect emerging retrieval challenges. Customers of work that is related to the Image Group have traditionally included U.S. security agencies, in particular the FBI. The USA PATRIOT Act specified that NIST would be the government arbiter of identification technologies. Homeland security activities also involve DARPA, as well as the Department of Justice and the Department of State for enhanced border and visa security. Much of this work is pertinent to corporations in the security field, such as biometric and smart card vendors. The Visualization and Usability Group has both government and industry customers. The work on interactive systems evaluation is directly supported by ARDA, DARPA, and ARL, and the voting and health care initiatives will have a usability component. Some of these efforts will result in test collec- tions to support the larger research community. The work on usability reporting has a broader focus on supporting effective information sharing between producers and consumers of software products. The work performed by the IAD is generally relevant to the customers referred to above. The Rich Transcription project and the speaker recognition work appear to be fulfilling the expectations of their sponsors, DARPA and NASA, respectively. The Retrieval Group works directly with government sponsors of the AQUAINT, TIDES (Translingual Information Detection and Summarization), and DUC efforts to define appropriate evaluation frameworks and metrics. The Retrieval Group's expertise and experience in developing new evaluation frameworks is valued by government agencies and is critical in evaluating the success of new technologies. The effectiveness of current work related to homeland security cannot be measured yet. However, NIST biometrics contracts have been continued for years, an attestation that the customers believe that the group is producing relevant results. Division researchers have accumulated and tested large finger- print and face test sets; this work is particularly applicable to border and visa security activities. Test results must be communicated carefully. For example, NIST communication should clearly state the specifications under which border security results apply (e.g., as controlled visa photographs). It is also advisable to emphasize that any attempt to extrapolate the results to different specifications risks creating false security expectations. External interest in the Pervasive Computing effort appears to have leveled off, judging by the flat
OCR for page 270
270 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 attendance at this year's workshop (in contrast to large participant increases for most other IAD- sponsored workshops). Pervasive computing is a speculative research area that NIST management has encouraged, without direct sponsorship, as likely to be important in the future. The panel supports this strategy of entering a promising field early to nurture and influence its growth, and it encourages continued efforts to nurture this field by database creation and by providing infrastructure software rather than by developing component technologies. Application of the Smart Space testbed to real-time, high-throughput sensor data processing for the Chemical Science and Technology Laboratory and the Physics Laboratory at NIST demonstrates the pervasive computing foundation and provides opportuni- ties to learn more about this application. The work on methods for evaluating complex interactive systems for intelligence analysts and robotic interaction is being defined in collaboration with the sponsoring government agencies. The results of these evaluations will be used directly by the agencies to evaluate technologies developed by contractors and to select and deploy useful and usable systems. The Industry and Usability Reporting work is done in collaboration with industry. There has been strong industry participation in defining and using the CIF standards. The certification by ANSI and the ongoing ISO process suggest that standards will be more widely adopted in the future. The growing recognition of usability as a key component of software procurement is important. IAD's projects are also generally effective. There are several examples of good dissemination of products and information generated by the division. IAD scientists have contributed more than 50 technical publications and talks. IAD continues to sponsor many workshops, addressing such topics as automatic content extraction, MPEG, rich transcription, speaker recognition, document understanding, INCITS V2, pervasive computing, industry usability reporting, topic detection and tracking, and TREC. For most of these workshops, participation is rising. For instance, in its eleventh year, TREC has had its highest participation level, participation at the speaker recognition workshop has doubled since the previous year, and the Industry Usability Reporting project has a growing, largely corporate, member- ship currently at 250. The annual TREC, TDT, and DUC workshops are major forums for interaction among the informa- tion retrieval researchers from universities, industry, and the government. The proceedings of these workshops are published by NIST and publicly available on the Web. NIST researchers also publish summative evaluation findings in top information retrieval conferences. The Retrieval Group has led the information retrieval community to explore new directions in question answering, cross-language re- trieval, and summarization. TREC continues to provide the research community with rich test collec- tions and assessments of relevance and to work closely with researchers in industry and academia in order to tackle new retrieval challenges in a systematic fashion. TREC has found a balance between the stability needed to explore ideas in depth and to develop useful test collections and the evolution to encompass new directions and technology trends. Several evaluation efforts worldwide, started in the past few years, have been modeled after TREC and include IAD members on their steering committees (e.g., CLEF ECross-Language Evaluation Forum] for evaluating multilingual issues for European lan- ... . ~ . · ,, ~ lo, lo, quakes. NTCIR rNAGSIS Test Collection for Information Retrieval Svstemsl for evaluating retrieval · · A · ~ ~ TO ATE ~ ~ AT · . · . · ~ TO ~ . · ~ ~ '~ AT __ . · ~ ~ ~ ~ . - Issues In Asian languages, and 1Nb2L Llmtlatlve tor evaluation ot 2~1L Retrievals tor evaluating structured retrieval). The IUSR effort, driven by the division, has involved a multiyear collaboration with a large number of industry participants. Participation in the fifth IUSR workshop in 2002 had doubled since the previ- ous year, attesting to the growing recognition of the importance of software usability. The workshop helped highlight activities related to the internationalization of CIF and extensions to requirements and hardware, which the Visualization and Usability Group is promoting.
OCR for page 271
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 271 Standards activities are one of the primary means of disseminating IAD work. IAD staff participated this past year in INCITS technical committees on MPEG, biometrics, and IT access interfaces and in Industry Usability Reporting, the Web3D consortium (formed to create open standards for WEB3D [3 dimensional] specifications), and the World Wide Web consortium on synchronized multimedia and usability. Division Resources . . . The need for personnel (e.g., in homeland security efforts) in IAD has been partly met by the addition of consultants and students. The level of facilities and equipment resources has remained adequate for the division. Personnel levels have remained flat. The large effort to test biometrics requires new computing resources and massive data storage; it appears that these resources have gener- ally been provided in an adequate and timely manner. An exception is the Meeting Room Transcription project, which captures audio and video data at the rate of 70 gigabytes/hour of meeting. The project needs additional storage and a higher-speed local area network for transmitting data. For over a decade, IAD has had productive involvement with the FBI and other U.S. security agencies. This involvement has included work in fingerprint, face, and gait biometrics. The steady support of this work has ramped recently with the homeland security initiative, although the technical emphasis has not changed much. IAD has been able to meet the demands of homeland security with knowledge, expertise, and infrastructure (software and hardware) built up over the years. Because of this long-term commitment, the homeland security work has not changed the balance of near-term versus long-term programs in IAD. People and money resources have not been diverted from nonsecurity programs in the division. The customer mix and cooperative relationships of the division have under- gone positive changes. In the past, interest and support were mainly from the FBI; now a wider range of government agencies are supporting and referring to NIST recommendations on biometrics. In previous years, the panel saw little synergy between the biometrics advocacy work done in the Convergent Information Systems Division (CISD) and the biometrics test design and database compila- tion work done in IAD, though both groups were producing well. CISD facilitated standards, mainly for nongovernment biometrics vendors, and IAD produced databases, designed tests, and wrote standards, mainly for government law-enforcement groups. The new USA PATRIOT Act requirements have brought the activities of both divisions closer. IAD is now testing commercial face recognition systems and is having to deal with commercial interoperability standards, areas in which CISD is also involved. CISD is involved in capture device testing and standards, both of which IAD must deal with in its technology assessment. The two divisions are interacting, and further synergy might enable NIST to lead technology matters related to biometrics more broadly. The speech and text retrieval work in IAD has traditionally been of interest to industry. More recent text retrieval work in Arabic, ACQUAINT, and speaker verification is of special interest to the govern- ment agencies involved in homeland security. IAD has managed to maintain its work related to business while performing this homeland security-related work. SOFTWARE DIAGNOSTICS AND CONFORMANCE TESTING DIVISION Technical Merit The Software Diagnostics and Conformance Testing Division develops software testing tools and methods that improve quality, conformance to standards, and correctness. The division also participates
OCR for page 272
272 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 with industry in the development of emerging standards, often taking a leadership role. The division is organized in three groups: Standards and Conformance Testing, Software Quality, and Interoperability. The technical merit of the work done by all three groups continues to be high. The Standards and Conformance Testing Group develops conformance tests and reference imple- mentations and conducts research on better ways to do conformance testing. The primary area of focus for this group is conformance testing for XML. The group's role expanded significantly this past year beyond conformance tests for XML itself; it now includes related technologies such as ebXML regis- tries and messaging, XSL FO, XML Schema, and XSLT/Xpath, among others. This expanded responsi- bility is largely at the invitation of industry consortia and working groups, reflecting the group's outstanding track record of working collaboratively with industry, providing technical leadership and ~ · ~ r ~~ ~ ~ r A unbiased feedback, and tacliltatlng cooperation and coordination among companies. The Software Quality Group develops methods to automate software testing, develops software diagnostic tools, and conducts research in formal methods. Current projects include Automated Test Generation, Computer Forensics Tool Verification, and Health Care Information Systems. The Auto- mated Test Generation project, which is mature, is currently being transferred to industry organizations that develop mission-critical software, and current work by division staff is focused on industry requests that have arisen during the transfer process. The Computer Forensics Tools Verification project is intended to provide a measure of assurance that tools used in the investigation of computer-related crimes produce results that are both technically and legally valid. The current focus is on testing commercially available disk imaging tools and write-blockers. Test results are reported to manufactur- ers so that they can improve their products and to law enforcement agencies so that they have a basis for deciding how and when to use particular tools. Work on Health Care Information Systems is focused on helping design a new, Java-based computing environment for the Department of Veterans Affairs. The Interoperability Group works with other federal government agencies, with the voluntary standards community, and with industry to increase the use of publicly available standards in order to achieve and enhance Interoperability A primary role of this group is that of working with government groups, including the Federal CIO (Chief Information Officers) Council, to apply standards and to develop Interoperability tests for IT systems and products that cross several agencies. Current projects ~ ~ ~ _ ~ ~ nclucle the National Software xeterence Library and Smart carcl lnteroperanlllty. lne National ~ott- ware Reference Library provides a repository of known software, file profiles, and file signatures for use by law enforcement and other organizations in computer forensics investigations. In partnership with GSA in the Government Smart Card (GSC) program, the Smart Card Interoperability project provides standards and tests to accelerate the use of smart card technology, both within the federal government and in the private sector. Program Relevance and Effectiveness The Software Diagnostics and Conformance Testing Division supports users and providers of software by facilitating improvements in software quality and Interoperability The division develops products such as reference implementations and conformance test suites, provides technical leadership by chairing standards committees and participating in consortia, and lays the groundwork for overall advances in this field by researching improved methods of conformance testing. NIST's role as an active but neutral third party in standards processes, coupled with the outstanding quality of the conformance tests developed by the division, provides government and industry with a service that is both necessary and unique. The division leads several ITL-wide initiatives in health informatics that promise to be of significant
OCR for page 273
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 273 consequence to the health care delivery community. Foremost of these is a collaborative effort with the Health Level Seven (HL7) standards community. This effort is focused on the HL7 3.0 data model and messaging formats. Until the participation of NIST, there had been very little effort on conformance testing in the HL7 effort. The lack of conformance has been evident in many commercial and public- sector health information systems, despite national mandates to ensure interoperability. NIST's exper- tise and management of the process is important to the state of the national health care system. The division is beginning to explore how to provide guidance for verification of compliance with the Health Insurance Portability and Accountability Act of 1996. In its early stages, this effort will, if successful, have significant import for the health care community. The division generally maintains close connection and excellent relationships with its customers, which include industry consortia, private companies, and government agencies. Project goals and metrics are generally responsive to customer requirements, which leads to technically superior results and efficient use of the division's resources. Because project goals and metrics are established at the incep- tion of a project, projects are concluded in a timely manner. A number of key projects have both started and finished this past year, including XSLT, XSL-FO, and the ebXML registries. The division continues to support concluded projects when there is a specific industry request for assistance; this requires very little staff time but is critical for technology transfer. The Software Diagnostics and Conformance Testing Division has demonstrated impressive success at beginning and concluding projects at appropriate times. Given the wide range of standards activities currently under way, the division shows excellent judgment in selecting efforts where it can have the most significant impact. STATISTICAL ENGINEERING DIVISION Technical Merit The goal of the Statistical Engineering Division is to advance measurement science and technology through collaboration on NIST multidisciplinary research projects by the development of statistical methods for measurement and metrology and by the application of statistical methodology to the collection and analysis of data critical to NIST scientists and engineers and to U.S. commerce. The division wishes to establish itself as a principal resource for statistical expertise in metrology world- wide. The primary role of the Statistical Engineering Division is to support research projects in other divisions or laboratories, rather than to develop projects within its own division or directly with indus- trial partners. The division has an extensive list of interactions with other NIST laboratories. These interactions vary from short-term activities to extensive collaborations requiring the generation of new statistical methodology. In many cases, these efforts are playing a fundamental role in establishing statistical methodology, which can also benefit organizations beyond NIST. Current projects are con- ducted in the areas of Bayesian metrology, key comparisons, IT performance, process characterization, measurement services, new methods for metrology, Internet products, and special programs. Problems being addressed by the division include the measurement of uncertainty in Standard Reference Materi- als, support of ITL biometrics methods, the development of a statistical methodology for key compari- sons, and the characterization of high-speed optoelectronic devices. A long-term collaboration between the Statistical Engineering Division and KEEL in the study of the properties of dielectric material has produced new methods for characterizing dielectric materials based on measurements of permittivity and loss tangent. The Statistical Engineering Division is working
OCR for page 274
274 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 to develop appropriate statistical techniques for use in assessing methods for combining biometric information in real time. Work in the area of key comparisons involves interlaboratory comparison studies designed to help determine the degree of equivalence among national measurement standards. The comparisons are chosen by the consultative committees under the International Committee for Weights and Measures. The Statistical Engineering Division is the leading statistical presence working toward a unified statis- tical framework and to provide guidance to ensure accurate, efficient assessment of equivalence. The division is developing new methods to address problems of estimating a consensus mean and eliciting and handling Type B errors. Improved measurement of high-speed, next-generation optoelectronic devices (40 to 80 Gbps) is of significance to optical-fiber communications, wireless communication, and Ethernet networks. NIST is working to develop calibration methods for optical reference receivers to support industry characteriza- tion of the impulse and frequency response. The division's role has been to produce new methodology to minimize the effect of sources of error, such as time drift, time-base distortion, and timing jitter on the measurement of optical receivers. The division is expanding its work on Bayesian metrology. Given that Type B error often involves nonverifiable assessments and that posterior distributions are of interest to division clients, attention to this area is appropriate. The division should also consider expanding work in data mining, bioinformatics, and computational statistics. Program Relevance and Effectiveness The Statistical Engineering Division is providing high-quality statistical support to collaborators and clients throughout NIST. While most efforts are intra-NIST collaborations, the division has been involved in activities of direct benefit to customers outside NIST. It has been involved in the establish- ment of baseline data sets for use in the assessment and evaluation of the computational accuracy of statistical software. In the area of key comparisons, the division is working directly with CIPM, national metrology institutes, and regional metrology organizations to establish sound statistical principles for the determination of equivalence of national measurement standards. The division recognizes that the development of new methodology is a primary component of its role. Many problems require similar methods, such as methods for estimating a consensus mean from results of interlaboratory experiments, or methods for addressing the impact of sources of uncertainty. The division is conducting research retreats to focus on developing such methods to achieve integrated solutions. The division is developing a review article for the statistics research community with the aim of attracting new researchers to the important issues relevant to key comparisons. The division has pro- duced a variety of approaches to estimate a consensus mean and is considering ways to achieve a unified approach that incorporates the knowledge gained across different efforts. Moving research efforts toward more complete solutions and producing solutions in a form that is most easily transmitted to clients and the statistics community are commendable activities. The Statistical Engineering Division has increased its efforts to be actively involved in the publica- tion and dissemination of statistical knowledge. Nearly all division statisticians are either presenting results at research conferences or publishing in refereed journals. The division could increase its impact through presentations at Research I universities, participation in organizing conference technical pro- grams, and increased publication in premier statistical journals. At this time, only a few of the division's
OCR for page 275
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 275 researchers are publishing in such journals. This would help to reestablish NIST as a recognized leading statistical metrology organization and would help draw the attention of the statistics community to the leading problems in metrology. Increased visibility would also aid in the recruiting of new staff mem- bers. The outreach and education effort of the division is strong. It provides courses and resources such as the NIST/SEMATECH e-Handbook of Statistical Methods to the NIST community. This effort helps to promote new collaboration and consulting opportunities. Division Resources The distribution of services provided by the Statistical Engineering Division appears to be limited by the current size of the division. Since 1997, the division has undergone a major change in size it once included 30 members. The current size of 19.4 full-time employees makes it difficult to maintain timely interactions and to meet task demands. New staff members are needed to replace lost capabilities. The division needs to add senior appointments of individuals with strong records of research and of methodological and theoretical development. Additional staff will also be needed if the division ex- pands activities into areas such as data mining, bioinformatics, and hierarchical modeling and computa- tional techniques, which are increasingly important statistical tools. The division has been successful at fulfilling its consulting role and at identifying opportunities for interaction. Limited staff size, however, has reduced its ability to mentor young researchers and to promote national visibility through publication and society activities. When demands exceed resources, the Statistical Engineering Division should consider reducing the number of traditional services that it provides in order to reallocate resources to higher-priority areas. Such decisions should involve a careful assessment of the impact that the division is having on the projects it supports and the relative ranking of the projects within the ITL and NIST strategic plans. The morale of the division appears to be generally strong. The division director has introduced an effective professional growth and annual planning and reporting process to reinforce staff investment. A new group structure has also been developed to support intergroup efforts. The Statistical Engineering Division staff expressed general satisfaction with the available comput- ing resources and with facilities. One long-standing issue, however, is the location of the division at the NIST North campus. This is apparently viewed by many division staff members as a major barrier to more effective interaction with NIST scientists. Accidental interactions are virtually eliminated, and strong initiative is required to maintain loose collaborations. As a result, staff may be more likely to become involved in the data analytic stage of a problem than in its design phase, where their expertise can also have a strong impact on the desired final outcome. Young division researchers have limited interaction with potential clients and with sources of expertise; they can fall into a pattern of research that does not consider the full range of solutions. Creating a satellite office suite for shared use on the main campus should be considered, along with information technology solutions that can enhance communication. The integration of the Statistical Engineering Division into many of the ITL ventures could be stronger, including new ITL initiatives contributing to nanoscale measurement science and homeland security. For example, the biometrics efforts could take advantage of new criteria for combining infor- -O 1 O . . O . mation for improved matching capabilities. In addition, the division could substantially contribute, with appropriate experimental design and modeling support, in many experimental and simulation situations, including biometrics and computer security-related efforts.
OCR for page 276
276 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 More organized coordination, at higher administrative levels, of the Statistical Engineering Divi- sion's participation in projects would be beneficial and might help alleviate some of the lost participa- tion caused by the North Campus isolation. CONVERGENT INFORMATION SYSTEMS DIVISION Technical Merit The Convergent Information Systems Division (CISD) conducts research on, develops, and increas- ingly promotes integrated systems, architectures, applications, and infrastructure for the exchange, storage, and manifestation of digital content; it also explores their scalability, feasibility, and realization for new applications. CISD currently applies its competency in electronics, physics, computer science, and engineering to high-impact areas that include homeland security, health care, e-commerce, and knowledge management. The division addresses traditional computer science application challenges as well as those in consumer electronics. During the past year, the CISD continued to provide industry with standardization and testing services for the exchange, storage, and manifestation of digital content. The division has continued its work in the coordination and standardization of biometric technology and exploratory work in authen- tication, access control and rights management for digital content. The projects examined for this assessment had metrics and time lines associated with them; how- ever, the level of detail with which they were described varied across projects. This made it difficult to determine if the milestone dates were being met for all projects. In two cases there did not appear to be a documented deliverable for the project. The division should produce for each project, in a standardized format, a time line showing key expected deliverables. The work in the area of content packaging and formatting aids in the development and definition of interoperability guidelines for digital content types. This past year, the group completed a file format inventory; a file format study on temporal usage, directed at issues relating to a time-based and com- pressed full-motion image; and an ATP/file format testbed, directed at issues relating to documenting archive media and to the feasibility of transferring scientific, technical, and medical documents to commercial platforms using the e-book format. The group's Refreshable Tactile Graphic Display (e- book) project addresses the goal of providing access for visually impaired and blind persons to images in e-books and other electronic media by creating a new tactile display technology for viewing images by the sense of touch on a reusable surface. The work in the area of content encapsulation and decapsulation involves developing interoperable open standards. guidelines. and specifications for rights management of digital content. The croup's Internet Trust and Digital Rights Management (DRM) project is aimed at developing guidelines to support trust, assurance, and the use of digital objects in e-commerce. The project's emphasis has been on improving electronics interconnect manufacturing and on assisting the digital products and services market. In 2002, the group reviewed general DRM issues and held a workshop to foster dialogue on technology and policy. The group's Biometrics Systems Research and Biometrics Standards Develop- ment project assists industry and government in the evaluation and deployment of biometric technolo- gies by developing middleware standards and their reference implementations and by developing proto- types, test methods, test data, and evaluation techniques for multimodal biometrics technologies. The work on the quantum communication testbed includes constructing a measurement and stan- dards infrastructure for quantum information technology. The infrastructure contains the testbed, cali- bration, and development facilities open to the technical community. The testbed is intended to demon-
OCR for page 277
INFORMATION TECHNOLOGY LABORATORY: DIVISION REVIEWS 277 strafe quantum communication and cryptographic key distribution at a high data rate (1.25 GHz) and to enable wide-ranging experiments on both the physical- and network-layer aspects of quantum commu- nication system performance. This effort is an important part of the wider NIST quantum information program. The work in the area of content consumption and digital preservation (content storage) focuses on developing metrology methods, technologies, and standards for digital preservation. The effort includes developing methods for testing the reliability of writable CD/DVD disks, including measurement of the life expectancy of CD-R and DVD-R disks, with a goal of providing an unbiased testing methodology for writable optical media. The effort also includes developing methods to test the compatibility of several writable DVD standards for both consumer electronic and PC-software DVD platforms; it also involves developing procedures and guidelines for testing and managing the archival preservation of electronic records with conventional (magnetic) and optical systems. In its image quality standards and test corpora work, the group is developing measurements and standards for assessing the image quality of moving pictures to support multimedia operability. The project involves use of the division's moving image quality laboratory for the subjective and objective evaluation of image acquisition, processing, and display quality. Relevance and Effectiveness CISD's numerous customers include developers of information technology systems, financial orga- nizations, biometrics firms, and other government agencies. The division collaborates with other ITL divisions and across NIST laboratories. Examples include collaborative work across ITL in the areas of biometrics and trust management, with KEEL on image quality and ICM 82B, with CSTL on the SEA (Science and Energy Alliance) conference, and with the Physics Laboratory on the quantum testbed. Across its groups, the division is engaged in projects relevant to the NIST special focus areas. Work in digital preservation, data interchange, and biometrics supports the homeland defense and information knowledge management areas; work on file standards supports the health care area; and work on the quantum testbed supports the nanotechnology area. Almost all of the projects reviewed for this assessment are relevant to the general industry that the division serves, with the exception of the quantum project. Although the project's research is interest- ing, its applicability to customer needs is not clear, because of the limitations in range and quality of over-the-air quantum cryptography. The testbed does not appear capable of addressing some of the fundamental interference issues that must be resolved before quantum systems can be deployed. CISD has served industry as a catalyst primarily by widely disseminating its work. Staff members contributed approximately 16 publications and more than 30 technical presentations during the past year, and they facilitated several workshops. Their presentation on digital preservation is significant. The Library of Congress highlighted the NIST work in its plan for the National Digital Information and Infrastructure Preservation Program. The CISD has published the results of its DRM and file format studies, which, in conjunction with image comparison tools, are critical for the analysis of temporal playback and compression. The divi- sion has developed new equipment to support research and development in the areas of ima~e-aualitv analysis and biometrics. as well as new eauinment for quantum laboratories. · ~ ~ ~ ~ 1 1 1 The division has expanded its involvement with external organizations, including the Advanced Television Systems Committee, DVD Association, High Density Storage Association, Financial Ser- vices Technology Consortium, Biometrics Consortium, Open eBook Forum, International Committee for Information Standards, Society for Motion Picture and Television Engineers, and Optical Storage
OCR for page 278
278 AN ASSESSMENT OF THE NIST MEASUREMENT AND STANDARDS LABORATORIES: FY 2003 Technology Association. It has also completed impact studies for e-books, biometrics, and the digital TV application software environment (DASE) and has improved its dissemination of results to relevant industries by making its Web site more user-friendly and by using the Web site as a vehicle for access to new division products. With its customers, the division should conduct a regular review of progress against the time lines and deliverables for projects (discussed above) as one effective way of determining whether the pro- grams are relevant and whether they meet customer needs. Division Resources It appears that the CISD funding derived from ATP may disappear in FY 2004 (its FY 2002 level was $426,000; its FY 2003 level is $320,000~. This may affect the biometrics multimodal work, as well as other projects deriving significant funding from the ATP. The quality of division facilities has improved. With the infusion of infrastucture funding, equip- ment has been constantly improving; in general, it is fairly new and nearly state-of-the-art technology. The digital preservation work would benefit from updated DVD test and microscopy equipment; video quality tests would benefit from an updated screen and video cameras; and quantum technology testbeds, from a new oscilloscope. The digital preservation, quantum technology, and biometrics work would progress better with the addition of a full-time employee to each project. The panel's concern in the FY 2002 assessment about funding and staff shortfalls has been ad- dressed by the division's obtaining additional funds for the quantum work and closing a $450,000 shortfall by preparing successful funding proposals and by engaging in interdivisional work. In response to the panel's urging that needs arising from the large number of student and guest researchers be assessed, the division has maintained a 1-to-1 ratio of full-time staff to students in the past year and has selected guest researchers for their ability to be creative self-starters. The CISD has revised its business plan to a 2-year R&D plan that can more flexibly adjust to the shifting needs of industry and government.
Representative terms from entire chapter: