|
||||||||||||||||
|
||||||||||||||||
|
||||||||||||||||
|
| ||||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||||
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 16
3
Assessment of the Laboratory Divisions
The Information Technology Laboratory is a very strong scientific and technical
resource for the nation with a long history of making lasting contributions. The ITL is
composed of six divisions: Applied and Computational Mathematics, Advanced Network
Technologies, Computer Security, Information Access, Software and Systems, and
Statistical Engineering. The ITL’s staff resources total 562, comprising the following:
322 technical federal staff members (57 percent of the total), 176 technical guest
researchers and contractors (31 percent), 46 administrative staff (8 percent), 15 faculty (3
percent), and 3 postdoctoral researchers (1 percent). Of the ITL staff, 38 percent are
computer scientists, 20 percent are mathematicians or statisticians, 17 percent are
information technology specialists, 12 percent are engineers or physicists, 12 percent are
administrative and support personnel, and 1 percent are social scientists. The following
sections present summary assessments for each of the ITL divisions.
APPLIED AND COMPUTATIONAL MATHEMATICS DIVISION
The Applied and Computational Mathematics Division provides leadership within
NIST in the use of applied and computational mathematics to solve scientific and
engineering problems that arise in measurement science and related applications. The
division’s staff does research and development in mathematics and computation and
participates with NIST colleagues in peer-to-peer collaborations on NIST problems.
Further, the ACMD supports the national and worldwide science and engineering
communities by developing and disseminating mathematical reference data, software,
and related tools. The division works with internal and external groups to develop
standards, test procedures, reference implementations, and other measurement
technologies for scientific computation.
The ACMD is composed of four groups: Mathematical Analysis and Modeling,
Mathematical Software, Computing and Communications Theory, and High Performance
Computing and Visualization. The division has 57 federal employees; 51 are at the
Gaithersburg, Maryland, NIST campus, and 6 are at the Boulder, Colorado, NIST
campus. Three of the federal employees are postdoctoral researchers. The ACMD also
has 29 associates, of whom 23 are guest researchers, 1 is a postdoctoral researcher, and 5
are students.
In regard to the charge of the Director of NIST to assess the degree to which
laboratory programs in measurement science, standards, and services achieve their stated
objectives and fulfill the mission of the ITL: the division has an impressive array of
technical expertise, and it is attacking a wealth of research areas. The quality of the work
meets the computational accuracy needs of consumer laboratories, which vary widely
among applications (e.g., from accuracy sufficient for visualization to highly accurate
computations for atomic structure). In most areas, the work of the division is very good
and comparable to what one would find in top 10 to top 20 university
applied/computational mathematics programs. In some cases—for example, the Digital
16
OCR for page 17
Library of Mathematical Functions—the work is unique and without peer in the broader
community.
The work of the division is accomplished primarily through a culture of small
“atomic” collaborations common in the work of applied mathematics. Impressively,
members of the division have established effective collaborations with materials science
and physics researchers.
On the downside, however, the culture of small, atomic collaborations may be a
hindrance to attacking important areas that require the formation of collaborative teams
from across many disciplines. However, the DLMF work demonstrated a capacity to
form a larger team of collaborators than is typical of this division. This ability to form
larger, effective collaborations may, in the long run, bode well for ACMD’s ability to
cope with future multidisciplinary problem areas in which the ACMD will be called on to
participate.
In regard to the charge of the Director of NIST to assess the technical merits and
scientific caliber of the current laboratory programs relative to comparable programs
worldwide: There are areas in which the accomplishments and capabilities of the ACMD
are excellent and surpass the capabilities of comparable programs worldwide. The ability
of staff to engage with physicists and materials scientists to use mathematics and
computing to provide insight on experiments is among the most successful such efforts in
the world. The Digital Library of Mathematical Functions is a unique and enduring
accomplishment. The Quantum Communications Program seems to be among the best in
the field.
In other areas the level of technology in the division work, while still excellent, is
below the state-of-the-art level. That is, the work is excellent by academic standards in
terms of the development of novel mathematical methods but is below the state of the art
when measured against the work of the division’s peers in other mission agencies in
meeting the goals of providing mathematical modeling and simulation expertise to the
rest of their organizations. For example, in the area of large-scale partial differential
equation (PDE) simulation, capabilities are mostly (but not exclusively) two-dimensional.
In addition, the level of mathematical rigor varies widely across the division. Code
verification is now viewed as an essential component for any large-scale simulation
project. Very systematic convergence studies similar to those done in the Parallel
Hierarchical Adaptive MultiLevel (PHAML) project might be adopted in other areas as a
method of code verification.
As to the charge to assess the alignment between laboratory R&D efforts and
those services and other mission-critical deliverables for which that laboratory is
responsible: While the level of technical expertise in the ACMD is certainly sufficient to
meet upcoming challenges, some significant cultural problems in the program
composition and scientific culture in the division could inhibit the response of the ACMD
to the challenges that the division views as critical to meeting its future needs.
The challenges facing the ACMD are multiscale and multiphysics, involve
complex geometries and new applications, and are of necessity multidisciplinary. In
addition, computational science is facing a major disruptive change in technology. This
change is the transition to multiple computing cores on a single processor, for which
many of the methods for parallelization and software engineering developed over the past
15 years lead to very low performance. Since this is an architectural change on the level
of a single processor, it will affect scientific computing at all scales.
17
OCR for page 18
The response of the larger applied mathematics community to the vastly increased
complexity of these problems has been to form larger teams, sustained over long periods
of time, in order to span the broad range of intellectual capabilities and challenges
required to make a successful attack on such problems. This is being done at universities,
at funding agencies, and at other national laboratory systems. This kind of change has not
yet taken place in the ACMD, and in fact it runs contrary to the more traditional “small
science” model in applied mathematics that has served the ACMD well in meeting many
of its mission requirements. It is certainly possible for the ACMD to form larger teams,
as evidenced by the DLMF and (to some extent) by quantum communications projects.
However, if the problems that NIST is investigating or will investigate are multiscale and
multiphysics, then such an approach will need to become more widely used in the
ACMD. Such a change may well require the ITL and NIST management to reconsider
the role of the ACMD in the laboratory as well. Owing to resource constraints, it may be
necessary to focus on a set of technologies and applications that lead to a high degree of
synergy, possibly at the cost of reducing the range of stakeholders at NIST. Another
possible approach to meeting these challenges, particularly in the simulation software
area, would be to collaborate with the laboratories in other agencies that would bring
strengths complementary to those of the ACMD (such as its success in collaborating with
physical scientists) and with whom the division could share the responsibility of software
development. The ITL and the ACMD should consider strategies to change the scientific
culture of the ACMD to meet the increased challenges facing the division. In addition to
broader, multidisciplinary simulations, discrete simulations might also be an area in
which the ACMD can address important challenges that align with the NIST mission
(network standard, for example).
ADVANCED NETWORK TECHNOLOGIES DIVISION
The mission of the Advanced Network Technologies Division is to “provide the
networking industry with the best in test and measurement research.”22 Its goals are “to
improve the quality of emerging networking specifications and standards,” and “to
improve the quality, reliability, resilience, robustness, security, and interoperability of
networked systems.”23 The ANTD assists in the conception and development of national
priorities in which networking is an essential enabler, and additionally it proactively
participates in the implementation of those developments.
The ANTD is composed of two groups: the Emerging and Mobile Network
Technologies and the Internet and Scalable Systems Metrology groups. The ANTD has a
regular staff of 29, and there are 10 guest researchers.
The ANTD focus areas are core networking infrastructure, ad hoc networks, edge
networks, complex systems, and networks and applications (cloud computing, smart grid,
and health care). The division’s delivery mechanisms include specifications and
guidelines, models and prototypes, test and measurement tools, reference
implementations and demonstrations, journal and conference papers, workshops and
conferences, and standards activities.
22
Donna F. Dodson, ANTD and CSD, “Advanced Network Technologies Division,” presentation
to the panel, Gaithersburg, Maryland, March 21, 2011, p. 2.
23
Ibid., p. 3.
18
OCR for page 19
High-Level Observations and Recommendations
The Advanced Network Technologies Division is strong and healthy, exhibiting
an awareness of and competence in achieving its mission within the ITL and within NIST
as a whole. Morale and motivation across the division appear high. The ANTD leverages
its relatively small size by taking advantage of opportunities for collaboration, both
internally with other divisions and with external organizations. It understands its mission
and strives to balance various demands on its attention, including mandates, supporting
science, standards advocacy, and strategic collaborations. The division demonstrates an
aptitude for responding to externally imposed agendas, often under tight time lines with
limited resources, while attempting to maintain a solid scientific foundation.
Networking is a key enabling technology for many, if not all, of the high-priority
activities of the ITL and NIST in general. National innovation and industrial
competitiveness are fueled by a robust, secure, and accessible national infrastructure. The
ANTD is and should continue to be at the forefront of the national discussion on vital
issues, standards, and policies of networking and worldwide communications
infrastructures.
Since the previous NRC review in 2009, the ANTD has made significant progress
on that panel’s suggestion for expanding the division’s exposure in the greater
community outside of specific standards organizations. The ANTD has increased the
quantity and quality of its publications, including papers in conference proceedings and
journals that the panel noted as being of high quality, and a staff member received a best-
paper award. The ANTD also has widened its external collaborations, in some instances
with grants and sometimes by seeking out companies, universities, and other research
organizations with common interests.
Despite these observed accomplishments, the ANTD faces several challenges.
First, it needs to acquire without delay a full-time director to articulate ANTD’s values
and interests and to define effectively long-term directions for the division and its
projects. Second, as a key collaborator in many projects, the ANTD finds itself
understandably drawn into activities that address many national priorities; these activities
are not only aligned with NIST’s charter but also positively impact national policy and
the economy. However, as a consequence of so many commitments, the division seems to
be understaffed, and the researchers’ ability to balance short-term and long-term NIST
goals can be impaired. Ideally, near-term projects in areas such as smart grid and safety
networks can be developed into core competencies, aligned with long-term strategic goals
in these important areas.
Recommendations
Following are the recommendations of the panel with respect to the Advanced
Network Technologies Division:
1. The ITL should fill the position of chief of the ANTD with a permanent chief.
2. ANTD management should address the understaffing issue in the division,
and in particular it should ensure that there are adequate resources to handle
both the short- and long-term needs of the division.
19
OCR for page 20
3. ANTD management should create a strategic roadmap for the technical work
of the division. The roadmap should be useful in managing the division’s
resources and portfolio of activities.
4. In addition to its portfolio of important activities, the ANTD should embrace
the future importance of cloud computing and draw up a plan for ANTD’s
leading role in a multidivision initiative within the ITL.
Project-Specific Observations and Recommendations
Internet Infrastructure Protection
In the area of Internet infrastructure protection, the ANTD has an exemplary
portfolio that represents technical foundations, standards leadership, policy guidance, and
adoption facilitation. The ANTD group provided an exceptionally lucid and compelling
presentation for the panel, detailing the motivation, accomplishments, and impacts of its
outstanding multifaceted work in this area. Particularly in the area of standards—with
regard both to the definition of standards (e.g., in which group members have co-authored
key standards documents) and to their adoption (e.g., IPv6 and the .gov deployment of
the DNSSEC)—the group is at the forefront of the field. The group’s leadership here
results in part from its sustained and long-term efforts in the area. Developing formal
methods to prove security properties of the various systems under study might be a
worthwhile research topic; although this topic had been explored in the past, recent
developments may justify another look at it as a possible growth area.
Smart Grids
The area of smart grids has tremendous impact and importance, and it fits into
NIST’s mission for supporting infrastructures that increase U.S. competitiveness. This is
new work for the division, and consequently the current work is somewhat narrowly
focused and does not take advantage of the breadth of the possible challenges. This
narrow focus may be because the group is responding to specific external requests.
Networking researchers can and should play a significant role in helping to define and
evaluate future smart grid architectures, including and beyond the communications
aspects of smart grids. The networking field, with more than 40 years of combined
experience in developing a highly scalable, informationally decentralized network-of-
networks, may well be able to contribute architectural, measurement, management,
control, and security techniques to the smart grid.
Public Safety Communications
The efforts of the ANTD in public safety communications are aimed at
developing performance-analysis tools to evaluate deployment scenarios and to evaluate
new technologies. These efforts are in collaboration with a number of federal agencies,
including the Department of Homeland Security (DHS) and the National Institute for
Occupational Safety and Health (NIOSH), as well as the Institute for Telecommunication
Sciences in Boulder, Colorado. The activities thus far have been a conceptually
straightforward evaluation of wireless-network deployment scenarios with various
20
OCR for page 21
channel models, with existing radio technologies. The private sector has likely developed
similar (possibly proprietary) tools. Before embarking on a large-scale or long-term
activity in this area, an analysis of existing tools (both public and private) should be
conducted, and partnerships with any promising tool developers should be explored.
Body Area Networks
Body Area Networks is a cross-divisional program (involving collaborators from
the ANTD and the Applied and Computational Mathematics, Statistical Engineering,
Software and Systems, and Information Access divisions) with a number of outside
collaborators. The panel was briefed only on the project developing a radio-frequency
propagation model for implantable and wearable medical sensors. The group is making
solid progress on the development of the radio propagation model using detailed models
of the human body and on the demonstration of the use of these models to determine
channel performance. These models have been included in a modeling tool. Model
validation for body surface propagation has proven extremely valuable and has provided
important insights; additional validation and continued (and extended) collaboration with
external research groups will be valuable. Congratulations are in order for the group’s
receipt of the Institute of Electrical and Electronics Engineers Personal, Indoor and
Mobile Radio Communications Conference Best Paper Award.
Measurement Science for Complex Information Systems
The area of measurement science for complex information systems is an excellent
example of core competency development through multidisciplinary, multidivisional
research (e.g., involving researchers from the ANTD and the Statistical Engineering and
the Applied and Computational Mathematics divisions). This work addresses an
increasingly common challenge in networking (and, more generally, the emerging
interdisciplinary field of network science) in which complex, large-scale systems must be
evaluated over a high-dimensional parameter space. The interdisciplinary contributions
here make the whole greater than the sum of the parts. The connection to a specific
application (cloud computing in this case) and the choosing of model details (such as the
virtual machine types and hardware platform classes) based on an Amazon Elastic
Compute Cloud (EC2) data center, are beneficial. The transition from congestion control
to a topical new area while retaining all previously acquired knowledge is a
commendable accomplishment. This is a well-chosen area with strategic vision.
COMPUTER SECURITY DIVISION
The Computer Security Division is composed of three groups: Cryptographic
Technology, Systems and Emerging Technologies Security Research, and Security
Management and Assurance. The CSD’s mission, “[to] provide standards . . . to protect
our nation’s . . . information systems,”24 is being achieved exceptionally well. Generally
speaking, the division’s activity can be divided into two categories: (1) activities that
directly create and communicate mission-critical deliverables, and (2) basic research and
24
Donna F. Dodson, ANTD and CSD, “NIST Computer Security Division,” presentation to the
panel, Gaithersburg, Maryland, March 21, 2011, p. 2.
21
OCR for page 22
development activities that support the identification, improvement, and creation of those
mission deliverables.
Mission-critical deliverables include a broad range of standards (including
cryptographic standards), technical and management guidelines, validation, testing, and
measurement tools, and technical reference materials and specifications to support
agency implementations. As a shining example, the NIST Special Publication 800* series
is renowned for providing technically sound, unbiased, relevant guidelines that are
frequently adopted voluntarily in private-sector procurements and practices and often
mandated by the OMB for use by the federal government. It is generally agreed that
NIST security guidelines are consistently superior to guidelines from other sources,
which are often vague, unrealistic, incomplete, or idiosyncratic.
The division program includes the following: coordination of large new efforts
such as the cryptographic hash competition and the Security Content Automation
Protocol Program; independent good judgment as exercised in the specification of the
transition to new key lengths; the development of conceptual taxonomies that structure
research in new fields such as cloud computing; the development of validation of
products, tools, and techniques through tools and trained outsiders; and outreach to
agencies, industry, and academia through NIST-managed workshops, external
conferences, and government committee participation. Given the broad scope of the
publication series, some publications will need updating soon; the authors are aware of
this need. The National Vulnerability Database (NVD) is another example of an
outstanding collaborative effort that is widely used and appreciated across the federal
government and by U.S. business. Efforts in the important area of authentication, such as
the study of biometrics risks and participation in the public-private NSTIC digital identity
initiative, tackle a messy, urgent problem for which no fully satisfactory solution is ready
yet.
Research and development activities explore basic science and engineering that
will sustain the division’s ability to create high-quality deliverables into the future as new
challenges emerge. Several of these projects are closely related to the division’s programs
in standards and technical guidance. For example, the work on combinatorial circuit
analysis and on side channel and differential power analysis is related to the division’s
work on cryptographic module validation. Such research work is important in that it
provides the division with the technical know-how required to evaluate modules or
components that implement cryptography.
The merit and caliber of the division’s mission deliverables are widely
acknowledged as being world-leading. Historical examples of this leadership include the
Advanced Encryption Standard (AES) introduced in 2001. The AES has been accepted
internationally and is an exemplar for conducting open competitions for cryptographic
standards. The CSD is at present conducting an international competition for a new
secure hash algorithm, the so-called SHA-3, and all indications are that the quality and
integrity of this process are at the same high levels. A significant number of other
security-related standards have been developed or are in process now (block cipher
modes, digital signatures, and key management, for example), and many influential
special publications (SPs) have been written (SP 800-108, Recommendation for Key
Derivation Using Pseudorandom Functions; SP 800-132, Recommendation for Password-
Based Key Derivation, Part 1: Storage Applications; and SP 800-38B, Recommendation
for Block-Cipher Modes of Operation: The CMAC Mode for Authentication).
22
OCR for page 23
Security tools such as the National Vulnerability Database have become highly
regarded international resources that are critical to modern security engineering. These
CSD activities are singular and are setting global standards for quality. Their value and
success have led others to create derivative products (sometimes not as high quality) that
may confuse users about the source of the products. Accordingly, NIST and the CSD
should protect their brand in this space as much as possible. For example, developing
quantitative analysis techniques and tools that define and estimate metrics from the NVD
would be one way to add value and to further NIST’s branding in this area.
The merit and caliber of the division’s basic R&D activities are comparable to
those of its peer, mission-oriented government laboratories such as those operated by the
Department of Energy, Department of Defense (DoD), National Aeronautics and Space
Administration, and National Institutes of Health. R&D activities of the division that are
most closely aligned with current and future NIST mission deliverables include work in
biometrics, cloud computing, virtualization, secure boot processes, test coverage
efficiency, and mobile platforms. Those activities are of high quality and are comparable
to applied academic, industrial, and government laboratory efforts.
There appears to be solid alignment between the division’s R&D efforts and its
mission-critical deliverables. When a research project is being proposed and created, it
will be leading edge to some degree—ahead of the specific standards and guidelines that
it is intended to support. Such is inevitably the case: research has to run far enough ahead
of the work on standards and guidelines to provide meaningful technical input. However,
this constraint means that any given research program may prove to be inapplicable to
technical standards, either because the research does not succeed or because the
directions of the market, standards, and technology diverge from the direction of the
research effort. As a general principle, given constraints on resources and the dynamic
nature of IT security technology, the division should be mindful of the relevance of its
research projects to the remainder of its mission and should be willing to sunset projects
in those cases in which the project has begun to achieve industrial or commercial success
or the focus of the project has diverged from the mainstream direction of information
technology or from the division’s work on standards and guidelines. The Role Based
Access Control Program appears to have achieved a measure of industrial success and is
perhaps a candidate for handing off to industry.
A recommended new initiative for the division would be to undertake a formal
collaboration with the Software and Systems Division in the area of secure software
development. Application-level vulnerabilities are a growing focus of hostile attacks in
both locally hosted and cloud computing environments. It is important that NIST bring
together its resources in security and software development to contribute in this area and
to prepare for the development of standards and guidelines that will help to raise the level
of the nation’s security in this important area. The ITL should consider creating a
collaborative effort between the CSD and the SSD that would be responsible for the
creation of standards and guidelines on secure software and enterprise-level system
development for use by government, industry, and academia. The CSD is also
encouraged to investigate, as appropriate, NIST’s role in metrics and guidelines for
privacy, a subject that was not specifically presented to the panel.
In sum, this well-managed division has a depth of expertise that contributes to
task forces defending the United States against advanced cyberthreats. This contribution
is particularly important given the ongoing national and international attention to the
23
OCR for page 24
cybersecurity challenge. Overall, staff in this division are spread thin, but they seem to be
juggling priorities well. The morale in the division appears to be quite high, creating a
positive work environment that has attracted top talent. These two facts—thinly spread
resources and a history of attracting top talent—suggest that the division should consider
hiring as a priority.
INFORMATION ACCESS DIVISION
Organization and Operation
The mission of the Information Access Division is to conduct research of
performance metrics and guidelines, standards and testbeds, and community-wide
evaluations in order to support the development of advanced “technologies for intuitive,
efficient, and effective access, manipulation, analysis and exchange of complex
information.”25 The division’s projected benefits of its work include the expedited
development of these technologies and their transition into the commercial marketplace
and into government applications.
The IAD consists of five major groups: Retrieval, Image, Multimodal
Information, Visualization and Usability, and Digital Media. The IAD also pursues four
crosscutting program areas: biometrics technology, human language technology, human-
system interaction, and multimedia technology. The division’s staff consists of 66
permanent employees, 12 contractors, 14 guest researchers, and 7 students—the IAD is
the largest division in the ITL in terms of staffing.
The educational background of the personnel is roughly as follows: one-half has
master’s degrees, and one-quarter each has doctoral or bachelor’s degrees. The range of
technical activities covered by these five groups is diverse, yet they can be generally
described and broadly grouped into types of activities as follows.
Biometrics: The use of biometric features for personal identification,
particularly related to face, fingerprint, iris, voice, and, more recently,
multimodality, which uses a combination of features for the same purpose;
Retrieval (and search): The retrieval of information in the form of image, text,
video, and three-dimensional objects, in response to user queries;
Human language technology: Community-wide technology evaluations that
assess progress in speech and language for text analysis, machine translation,
speech analysis, and speaker and language recognition;
Visualization and usability: The study of usability by humans related to
specific implements and systems such as electronic voting machines, health
information records, and biometric devices; and
Multimedia technology: Evaluations, standards, and challenge problems in
video analysis, digital data preservation, three-dimensional imaging, and
biomedical imaging.
25
Curt Barker and Leslie Collica, ITL and IAD, “Information Access Division (IAD),”
presentation to the panel, Gaithersburg, Maryland, March 21, 2011, p. 5.
24
OCR for page 25
The biometrics and retrieval activities enjoy a long track record of
accomplishments. The usability initiative is emerging and showing the potential of
important impacts. The video analytics, multimedia event detection, and the retail
surveillance projects exemplify emerging projects of smaller scales.
The funding structure of the division is unique among ITL divisions. Sixty
percent of the division’s budget comes from OA funding (other agencies, such as from
the DHS, DoD, FBI, and Intelligence Advanced Research Projects Activity), which is
mostly security-related. Only about a quarter of the budget is classified as IAD base
funding, and less than 15 percent is from other sources at NIST.
The output of the work in the division includes the following:
1. Standards and guidelines: Defining the best practice along a specified
dimension or aspect of the relevant technology development;
2. Data repository: Acquiring, verifying, organizing, and supporting the various
important standard data sets required for developing and evaluating the
relevant technologies;
3. Evaluation and benchmark results: Conducting and reporting impartial
community-wide evaluation results to help gauge the progress in the relevant
technology;
4. Outreach: Organizing workshops (some, e.g., the Text Retrieval Conference
[TREC],26 have been going on annually for almost 20 years) to present
findings of the technology evaluations that bring together people from
academia, industry, and government; hosting students and guest researchers;
and
5. Reports, publications, and software: Publishing and archiving relevant results
and reports; providing benchmark software (e.g., fingerprint image quality and
matching).
Division Accomplishments and Observations by the Panel
The IAD is responsible for meeting a number of high-value and critical national
priorities and mandates. Several groups in the division are uniquely qualified to address
these mandates. Many scientists in the IAD continue to play a major role commendably.
Examples of these mandates include the USA PATRIOT Act, the Enhanced Border
Security and Visa Entry Reform Act of 2002, the Help America Vote Act of 2002, and
the Health Information Technology for Economic and Clinical Health Act of 2009.
Within the division’s charter, the groups demonstrate a high level of technical
competence in conducting long-standing, large-scale technology evaluations in critical
technologies that are extremely important to the government, business, and academia, not
only in the United States but worldwide. Examples include open community evaluations,
such as TREC, and large-scale biometric testing (e.g., fingerprint, face, iris, and speech).
Given the staff’s expertise and its neutral role, NIST evaluations conducted by the IAD
are considered fair and beneficial to the sponsors and to technology developers, and they
foster research in academia.
26
TREC is a forum for evaluating technologies for detecting, searching, and retrieving query text
in a large set of documents.
25
OCR for page 26
The division appears to take a proactive role in identifying new directions for
pursuing its mission and is adapting well to meet the national mandates and progress in
technology. Examples include the progression of the Retrieval Group from the retrieval
of spoken documents in TREC to video retrieval in TRECVid (TREC for video
“documents”). The Image Group started out mainly in fingerprint matching and
evaluation, but it now has broadened its expertise and conducts large-scale evaluations in
face, iris, and multimodal biometrics. Examples of some other new initiatives include the
evaluation of computerized tomography (CT) lung image analysis and usability studies
for biometrics.
The IAD demonstrates unique capabilities in establishing biometric data standards
and large-scale performance evaluation benchmarks.
Recommendations
Following are the recommendations of the panel with respect to the Information
Access Division:
1. The Information Access Division supports the development of technologies
and their transition into the commercial marketplace as well as government
applications. The division currently relies on substantial and sustained
amounts of other agency (OA) funding (approximately 60 percent of IAD
funding). Most of the OA funding is security-related (from the Department of
Homeland Security, the Department of Defense, the Federal Bureau of
Investigation, and the Intelligence Advanced Research Projects Activity). The
reports, standards, and evaluation studies of the IAD are closely followed by
academia and industry. In light of increasing foreign dominance of the
biometric industry, IAD’s reliance on OA funding, and IAD’s work in support
of biometrics technology development, it is important that the IAD and the
ITL remain mindful of the NIST mission to promote U.S. innovation and
industrial competitiveness, and so IAD efforts should continue to place
highest priority on the needs of the nation’s commerce even while pursuing
activities involving international sponsors.
2. The IAD should establish a long-term vision and identify some fundamental
research problems (e.g., relevant measurement science) that it determines
should be pursued in addition to the ongoing activities.
3. The Image Group should highlight its work in biometrics and differentiate its
work from that of other groups in the IAD that also deal with images.
4. The NIST administration should initiate a review of the interagency
negotiation process, joined by the individual and relevant external agencies if
necessary, to reduce the unnecessary administrative impediments that may
adversely affect the technical progress and jeopardize future contracts.
5. The ITL should work with the NIST administration, and possibly the policy-
defining organization, that is, the Department of Commerce, in reviewing the
institutional approval process related to the Institutional Review Board (IRB)
to maximize efficiency and minimize unnecessary latency.
6. The IAD should enhance outreach and external interaction as well as
personnel professional development. For example, a NIST-wide internship
26
OCR for page 27
program at the graduate level with a tracking mechanism may prove helpful in
raising the institutional image, the promotion of IAD activities, and the
recruitment of new talents.
7. The IAD should review its recruiting policy and practice so as to ensure a
healthy influx of new and young talent.
SOFTWARE AND SYSTEMS DIVISION
The mission of the Software and Systems Division is “to accelerate the
development and adoption of correct, reliable, testable software leading to increased trust
and confidence in deployed software, systems, and products.”27 Four research groups
comprise the SSD: Software Components, Information Systems, Interoperability, and
Electronic Information. The SSD recently hired a new division chief and two managers.
The division staff performs high-quality work and is having good impact on a number of
high-visibility, national priority projects.
On the question of the degree to which the division’s programs in measurement,
standards, and services achieve their objectives: The SSD work to date in software testing
and conformance is strong, although the SSD should look to the future and position the
division to get measurement, standards, and data sets out ahead of urgent needs.
Regarding the overall merits and caliber of the program: The SSD efforts in niche areas
are of unquestioned leadership, and in general the quality of work is comparable to that of
its peers in any other government or private organization. In addition, inter- and
intralaboratory collaborations are a key strength of several members of the SSD group.
On the question of the alignment of division R&D efforts with its mission, the work in
the SSD is well aligned with the missions of the SSD, the ITL, and NIST; however,
application-level projects could be better leveraged such that the methods, techniques,
and tools developed for one project are readily available for reuse in new projects. It is,
of course, clear that customer needs must be met, but a continuing focus on the potential
generalizability of the technologies developed within the context of a particular project
will ensure that the SSD staff is ready to take on new projects more quickly and
effectively.
Formal Performance Metrics
Foundational work can be challenging to measure objectively. One possible
method for measuring the external impact of foundational work might be through
applications that are built on the core methods and techniques. For example, the merits
of software testing could be measured by assessing the improvements in voting systems
subsequent to their having undergone rigorous NIST-based testing. Measuring
applications should lead to increased emphasis on customer-oriented metrics. For
example, Software Assurance Metrics and Tool Evaluation (SAMATE)-related
improvement in static analysis tools (perhaps as measured at Static Analysis Tool
Exposition [SATE] events) and overall quality of code (perhaps as measured by vendors
of software testing tools) could augment more internal, process, or local metrics.
27
Ram D. Sriram, SSD, “Software and Systems Division,” presentation to the panel, Gaithersburg,
Maryland, March 21, 2011, p. 8.
27
OCR for page 28
Strategic Plan
The new SSD leadership is in the early stages of developing a strategic plan. To
accelerate the process, the leadership identified an outside expert who will work with it to
help define goals and objectives. As part of developing the strategic plan, the group
should re-examine its mission statement and should consider the relative benefits of
aligning the SSD technical organizational structure with foundational methods rather than
with specific applications. As projects come and go, the organization will thus be better
positioned to bring together the appropriate expertise for new projects as they arise. Once
such a new organizational structure is in place, it should become clear where there are
gaps in expertise, and additional formally trained individuals in gap areas should be
hired.
Application-Level Projects
Externally mandated and funded projects are subject to sunsetting, and the team
should be ready to let go of those application areas that are no longer priorities.
Application-level projects are important opportunities to fulfill SSD’s mission, but it is
important that the division continue to build the core approaches, methods, and
technologies particularly so that, as new application-level projects arise, the group is well
positioned to address them.
Reference Data Sets
A core role of the ITL is to generate and curate reference data sets. Although this
role requires significant effort, other ITL divisions have been highly successful in such
efforts. For example, the ITL work on the TREC project has been exemplary, and it
would serve the SSD well if it were also to take a lead in generating, curating, and
maintaining reference data sets, including, if possible, those in the health domain.
SSD Research Groups
Software Components Group
The SSD should consider technology developments that have occurred and are
likely to occur in the first part of the 21st century, and it should reevaluate the activities
of the Software Components Group. The National Software Reference Library (NSRL)
for software for mobile devices is a good new effort. Additional energy should be
applied to improving and speeding up the process of generating and distributing the
reference library given the new, and now much more common, models of commercial
software distribution. Examples include auto-updates and application downloads,
software as a service, and cloud computing.
The software assurance projects such as SAMATE, SATE, and the SAMATE
Reference Data Set (SRD) are solid, appropriate SSD projects. More metrics related to
external impact should be used to evaluate the (likely very positive) impact of these
projects. In particular, the SSD could quantify impact by comparing old and new tools
against the same code base, and possibly it should gather statistics from tool vendors on
28
OCR for page 29
industry-wide improvements in software quality. On the SRD in particular, the SRD
code snippets are too small (100 lines of code [LOC]). The SSD should add larger code
examples—perhaps some from SATE—into the SRD (e.g., Emacs [~300K LOC C],
Chrome [~3M LOC], BSD network stack).
Information Systems Group
The voting work being conducted by the Information Systems Group is moving
apace. The group should consider the implications of the completion of the voting
project, given that the Election Assistance Commission may sunset this project. SSD
staff should aggressively publish on this work so that it, and, more generally, NIST,
receive appropriate credit for their contributions.
On bioinformatics and bioimaging, the group is collaborating well with other
laboratories at NIST. The SSD should capture best practices here, and they should be
replicated across the ITL. The medical imaging work has the potential for impact by
advancing the quality and consistency of measurement in clinical use. However, this
work and other SSD efforts in the biosciences are less well aligned with the SSD mission
than are some other SSD projects. If the SSD wishes to build strength here, it should
clarify how this project contributes to its core foundational work and then recruit domain
leadership in that area.
Interoperability Group
Test methods developed by the SSD for meaningful use certification have been
somewhat helpful to the Electronic Health Record (EHR) vendor community and the
Office of the National Coordinator for Health Information Technology (HIT). (This
work allows EHR vendors to appear on the Certified HIT Product List.) As the funding
from the American Recovery and Reinvestment Act of 2009 (Public Law 111-5) comes
to an end, work on testing infrastructure for health IT has the potential for use in new
application areas that the Interoperability Group should consider. The energy devoted by
the group to “tools for tools” is well placed. It is helpful for NIST to bridge and help
bring consistency across various interoperability groups (e.g., the Nationwide Health
Information Network [NwHIN] and Integrating the Healthcare Enterprise [IHE]); it is
nevertheless important to maintain both the perception and the reality of NIST neutrality,
particularly in the IHE involvement.
Electronic Information Group
The smart grid framework and roadmap work is a contribution to the community.
The specific work on 1588 time synchronization is interesting. The framework and
roadmap document for the smart grid released as a draft is likely to have positive impact
on the smart grid community and future industrial development.
29
OCR for page 30
STATISTICAL ENGINEERING DIVISION
The Statistical Engineering Division has a 65-year history of consistent and
fundamental contributions to the central mission of NIST through the development and
application of statistical methodology for metrology. This methodology supports the
basic NIST activities of producing measurements and quantifying their associated
uncertainties.
The SED advances its mission of supporting research in measurement science,
technology, and the production of standard reference materials (SRMs) through
numerous scientific collaborations within NIST and externally. The SED conducts
statistical research and provides important training and educational activities within and
beyond NIST. It serves as a unique national and international resource for the metrology
and standards communities and more broadly in high-profile contexts where an
acknowledged impartial broker of data analysis and interpretation is needed.
The SED is composed of two groups: one at the NIST campus in Gaithersburg,
Maryland, and the other at the NIST campus in Boulder, Colorado. The Gaithersburg
group has 20 technical staff members plus a support secretary; the Boulder group has 3
technical staff members and a secretary.
Achievement of Stated Objectives
The SED mission statement is that the SED “seeks to contribute to research in
information technology, to catalyze scientific and industrial experimentation, and to
improve communication of research results by working collaboratively with, and
developing effective statistical methods for, NIST scientists and our partners in
industry.”28 The division is highly successful in achieving these objectives outside of the
ITL, where its contributions (inside and outside of NIST) are highly regarded, effective,
and visible. The management and staff of the division also believe that there are
additional opportunities for genuinely substantial contributions to existing efforts inside
the ITL (as personnel limitations allow).
Inside NIST, the SED provides essential support for basic metrology efforts (for
example, in the production of standard reference materials). The division’s personnel
also have deep collaborations with NIST scientists in both traditional and emerging areas
of science and technology, ranging from neutron physics, to the large-scale measurement
of greenhouse gas concentrations, to basic signal processing, to the mapping of return
values of extreme wind gusts. The division’s lead role in the new NIST (Innovations in
Measurement Science) Shape Metrology Program is evidence of the quality of its work
and effectiveness in NIST collaborations.
Outside of NIST, the SED responds on an emergency and ad hoc basis to national
and international needs, is engaged on an ongoing basis with important U.S. researchers
and agencies, and contributes materially to international metrology groups. Recent
examples of important externally focused work of these types include the development of
an oil budget calculator for the Deepwater Horizon oil spill; work with the Office of Law
Enforcement Standards on body armor performance; collaborations with medical
researchers on computer tomography and optical medical imaging; long-term work with
28
See http://www.nist.gov/itl/sed/mission.cfm. Accessed July 11, 2011.
30
OCR for page 31
the U.S. Departments of Justice and Homeland Security and the Office of National Drug
Control Policy; membership on technical committees of the International Organization
for Standardization (ISO); contribution of basic material to new ISO standards
development; and invited participation in important international workshops on
mathematics and metrology.
The SED makes substantial educational contributions both inside and outside
NIST and effectively disseminates sound statistical methodology for metrology. Division
personnel are active in the development and delivery of metrology-related statistics short
courses from Gaithersburg to South America and Europe. The division maintains and has
recently updated the online Engineering Statistics Handbook,29and it develops and
disseminates specialized software supporting best practices in statistical metrology. It
has an excellent record of refereed publications, both papers done jointly with subject-
matter scientists in fields of collaboration, and publications in the statistical (and
statistical metrology) literature.
The most obvious opportunity for the SED to have additional impact is within the
ITL itself. If the fairly severe need for a larger PhD-level staff can be alleviated, a
number of emphases in the ITL could be catalyzed by more involvement of the SED.
These possibilities include work in data management and visualization, work in pattern
recognition and machine learning, and the laboratory’s work in forensics. The SED must
be a central contributor to NIST work in forensics to the extent that the Daubert
requirement30 for the evaluation and statement of potential error rates is to be upheld. The
situation is not unlike SED’s role in developing quantifications of measurement
uncertainties. While there is already some division activity in this area, notably in the
National Ballistics Comparison Project, much more is possible and needed as personnel
are available.
Technical Merits and Caliber of Work
The SED work is technically excellent and central to the overall mission of NIST.
The division provides state-of-the-art support for traditional NIST activities such as the
production of standard reference materials, calibration studies, and assessments of
interlaboratory studies. It is an innovative partner in the development of complicated
emerging measurement technologies like high-speed opto-electronic measurement. Its
external reputation for excellence is easily established through a consideration of its
solicitation to work on high-profile, pressing national and international issues such as the
Deepwater Horizon oil spill disaster, the estimation of Colombian coca cultivation, and
law enforcement body armor reliability; its steady stream of internal NIST and external
awards (including three Department of Commerce Gold Medals in 2010 and the Youden
Award from the American Statistical Association in 2009); its record of peer-reviewed
publications in statistics, metrology, and basic sciences (which are strong in terms of both
the venues used for dissemination and the number of papers published); and invitations to
29
Available at www.itl.nist.gov/div898/handbook. Accessed August 15, 2011.
30
The reference is to the 1993 Daubert v. Merrell Dow Pharmaceuticals, Inc., U.S. Supreme
Court decision that set current rules and requirements for the admissibility of expert scientific testimony in
federal courts. Among other things it requires that error rates of methods used to support expert scientific
testimony be known and stated.
31
OCR for page 32
international service, both on society committees and in the providing of training in
statistical best practice.
Alignment of Research with Laboratory Services and Deliverables
The alignment of the SED research with ITL services and deliverables is
excellent. The SED projects and participations fully contribute to the ITL roles of
advancing science and engineering, setting standards and requirements for unique
scientific instrumentation and experiments, data, and communications. It is the very
nature of good statistical research that it is driven by the real data-analysis needs of a
substantive field. The SED research is excellent and thus both inherently and effectively
collaborative in nature (inside and outside NIST).
Developments Since the 2007 and 2009 NRC Assessments
The SED management continues to be excellent and gaining momentum. It is
proactive in finding and defining important projects and in aligning the division’s work
with ITL and NIST priorities. It is proactive in marshaling appropriate funding and
promoting international linkages. The leadership is technically active and effective, and
the group morale appears to be high. The number and visibility of projects (including the
new NIST Innovations in Measurement Science project) are increasing.
Two strong hires have been made since the previous NRC review, but two
compensating retirements have prevented the growth in the number of PhDs strongly
recommended in both the 200731 and 200932 reviews. There have been no new
partnerships established with academicians and no division postdoctoral researchers since
the 2009 review.
Division Needs
Increasing the number of PhDs in the SED is a critical need (identified, as noted
above, in the 2007 and 2009 NRC reviews). This is particularly true in light of potential
retirements and the exponential explosion of the complexity of statistical and
measurement methodologies, and of NIST needs (related, e.g., to forensics work). At
least one hire per year for the next 5 years is needed in order to cover both imminent
retirements and any other attrition and to provide for growth of at least a 2 to 3 full-time-
equivalent staff in the next 5 years in order to handle the increasing complexity and
volume of SED involvement in important new initiatives and areas. In addition, an
increase in visiting faculty and other scientists is needed to help foster long-term external
collaborations. More than one-time events and/or visits and a much broader scope here
would allow the division to leverage its trove of excellent problems and opportunities to
31
National Research Council, An Assessment of the National Institute of Standards and
Technology Information Technology Laboratory: Fiscal Year 2007. Washington, D.C.: The National
Academies Press, 2007.
32
National Research Council, An Assessment of the National Institute of Standards and
Technology Information Technology Laboratory: Fiscal Year 2009. Washington, D.C.: The National
Academies Press, 2009.
32
OCR for page 33
increase not only its own output, but the quality and relevance of U.S. academic
statistical research.
For the direct development of the SED staff, increased flexibility, support, and
opportunity for visits by SED personnel to academic and non-academic institutions to
establish long-term collaboration would be beneficial. So also would increased funding
opportunities and support for participation in conferences and other professional
statistical activities. The latter is important not only for the development of existing staff,
but also to make the division attractive to the best new potential hires.
Opportunities for the Future
Overall, the situation in the SED is quite healthy. Important and fascinating
opportunities outstrip the division’s present physical ability to respond. There is much
important work going on within the ITL that could genuinely benefit from increased SED
involvement and collaboration. If the small group in Boulder were larger, there would be
the opportunity to collaborate with scientists at the National Center for Atmospheric
Research and the National Oceanic and Atmospheric Administration. More could be
done in the way of participation on editorial boards and the organization of sessions (at
both statistical and metrological meetings). More could also be done to raise the SED
profile in the statistics community. The division has access to some of the most
interesting statistical problems in the world, and it does some of the most interesting
physical-science-related work in the world. Statisticians broadly should know more
about opportunities and the work at NIST.
33
