Appendix I
Sample Crosscutting Assessments
As the nation’s measurement laboratory, the National Institute of Standards and Technology (NIST) has always had the mission of working with industry to provide measurement techniques, data, and reference materials and to work with standards-setting organizations to provide the technical expertise required to assist them in the establishment of standards. Most of this activity has been carried out by small groups within the NIST laboratories, working with external partners in other government agencies, at universities, and in individual or collective groups of industrial organizations. The enormous array of these programs has always made it somewhat difficult for external assessments to be comprehensive.
Driven for the past several years by many external factors including the strong focus of the current administration on industrial competitiveness and manufacturing, NIST has been engaged in aligning its programs to fit these broad areas of government emphasis. Advanced manufacturing, as defined in the 2012 National Science and Technology Council (NSTC) document A National Strategic Plan for Advanced Manufacturing,1 is impacted by already existing and newly initiated activities within NIST, with participation in one manner or another by all NIST organizational units. In 2011 the Director of NIST asked the National Research Council (NRC) of the National Academies to assemble a panel to assess the state of this vast enterprise within NIST.
NIST organized its program presentations for the NRC panel established to carry out the assessment to highlight three areas: Nanomanufacturing, Advanced Materials, and Smart Manufacturing. The NRC panel included members with expertise in the technical areas covered and, in many instances, having previous experience with NIST programs through direct interaction or in the course of service on an earlier NRC assessment. Although the presentations made to the panel were focused primarily on program scope and organization, some examples of actual past and current technical work were covered in sufficient detail to allow judgments to be made about program quality and the caliber of personnel.
It is too early to judge whether this exercise in comprehensive review of crosscutting program content will lead to significant impact on the NIST program. Nonetheless, the very fact of participation in such a review encouraged NIST presenters at all levels to think carefully about what they would say to outsiders. This internal, thoughtful assessment by capable people who care deeply about their programs and the reputation of the organization and of its staff can be the most important outcome of any assessment.
In a second example of a cross-organizational assessment, an NRC team of experts conducted a crosscutting review of the Army Research Laboratory’s (ARL’s) Autonomous Systems Program. Robotic systems have become ubiquitous in military operations, entering first as devices controlled from nearby by a single individual and now appearing as swarms controlled from a distance. With a vision toward fully autonomous systems with increasing capabilities, all of the military services and the Defense Advanced Research Projects Agency have launched extensive research and development (R&D) programs. Throughout the ARL directorates,
____________________________
1 National Science and Technology Council, 2012. A National Strategic Plan for Advanced Manufacturing. National Science and Technology Council, Washington, D.C.
programs have developed that focus primarily with respect to thrusts on materials, power sources on propulsion, protection, sensing, and human interfacing. Recognizing that results from these programs must ultimately be integrated to yield a final system, cross-organizational efforts have been launched, and the area of autonomous systems has been elevated to the level of senior-level visibility. This effort represents a very large component of the ARL science and technology program, yet it grew from many, rather separate, local programs and then was wrapped up for coordination. Many of these programs have been maturing for years. In some respects this ARL Autonomous Systems Program resembles in kind what might be found in the NIST Advanced Manufacturing Program several years from now. How is the Autonomous Systems Program at ARL progressing? What is the technical quality of the work being done? Are appropriate areas being covered? Are required cross-disciplinary issues being addressed? Those were the challenging questions posed by the ARL to the ARL Technical Advisory Board (ARLTAB) of the NRC.
The ARLTAB is charged with regular assessments of the technical quality of the work done at the ARL. Peer panels of experts visit the roughly disciplinary directorates of the ARL on an annual basis and view most of the ongoing program over a 2-year period. To address the onetime charge of assessing the Autonomous Systems Program, the NRC identified technical experts from the various existing panels and assembled them into a large panel with the range of skills covered by the program description. The panel heard overview talks presented by leaders of the various thrusts and integrated programs. They formed smaller groups and held simultaneous sessions, which were generally organized around posters. They had an opportunity to meet directly with scientists and engineers to get the pulse of current activity.
This challenging set of presentations and the format of the review created a first-time opportunity for program self-evaluation. Representatives of some parts of the program were learning more about other parts. This learning vehicle proved useful for researchers in a large, loosely connected set of R&D programs in separate buildings on two campuses. One important advantage in such a review is being able to assemble the panel from a large cadre of experts, each of whom has familiarity with the global mission of the laboratory and an in-depth knowledge of part of the laboratory from previous service on a disciplinary peer-review panel.
Both examples of cross-organizational programs cited above represented R&D within a large laboratory structured mainly along disciplinary lines. They spanned the time frame from program inception to maturity. Matrix management is not the overarching structure in the organizations cited, nor are most efforts cross-organizational. However, in each instance cited, important aspects of program assessment would not have occurred through a process organized according to disciplines. Also noteworthy, in each case cited, is that the crosscutting effort benefited significantly from the establishment of the peer group from a large cadre of experts who were already fully engaged in disciplinary assessment of elements of the cited programs. Panels organized for assessment of technical directorates are an ideal source of expertise for specialized, ad hoc panels targeted at cross-organizational issues.
