Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 12
1 Introduction This introductory chapter describes the biennial assessment process conducted by the National Research Council’s (NRC’s) Army Research Laboratory Technical Assessment Board (ARLTAB). It then describes the preparation and organization of the report, the assessment criteria, and the approach taken during the report preparation. THE BIENNIAL ASSESSMENT PROCESS The charge of ARLTAB is to provide biennial assessments of the scientific and technical quality of the Army Research Laboratory (ARL). These assessments include the development of findings and recommendations related to the quality of ARL’s research, development, and analysis programs. The Board is charged to review the work of ARL’s six directorates but not to review two key elements of the ARL organization that manage and support basic research: the Army Research Office (ARO) and the Collaborative Technology Alliances (CTAs). Although the primary role of the Board is to provide peer assessment, it may also offer advice on related matters when requested to do so by the ARL Director; such advice focuses on technical rather than programmatic considerations. The Board is assisted by six NRC panels that focus on particular portions of the ARL program. The Board’s assessments are com - missioned by ARL itself rather than by one of its parent organizations. For this assessment, ARLTAB consisted of seven leading scientists and engineers whose experi - ence collectively spans the major topics within the scope of ARL. Six panels, one for each of ARL’s directorates,1 report to the Board. Six of the Board members serve as panel chairs. The panels range in 1The six ARL directorates are the Computational and Information Sciences Directorate (CISD), Human Research and Engi - neering Directorate (HRED), Sensors and Electron Devices Directorate (SEDD), Survivability and Lethality Analysis Director- ate (SLAD), Vehicle Technology Directorate (VTD), and Weapons and Materials Research Directorate (WMRD). The Board 12

OCR for page 12
13 INTRODUCTION size from 10 to 20 members, whose expertise is carefully matched to the technical fields covered by the directorate(s) that they review. In total, 96 experts participated, without compensation, in the process that led to this report. The Board and panels are appointed by the National Research Council with an eye to assembling balanced slates of experts without conflicts of interest and with balanced perspectives. The 96 experts include current and former executives and research staff from industrial research and development (R&D) laboratories, leading academic researchers, and staff from Department of Energy national laboratories and federally funded R&D centers. Twenty-eight of them are members of the National Academy of Engineering, 4 are members of the National Academy of Sciences, and 1 is a member of the Institute of Medicine. A number have been leaders in relevant professional societies, and several are past mem - bers of organizations such as the Army Science Board and the Defense Science Board. The Board and its panels are supported by NRC staff, who interact with ARL on a continuing basis to ensure that the Board and panels receive the information that they need to carry out their assessments. Board and panel members serve for finite terms, generally 4 to 6 years, staggered so that there is regular turnover and a refreshing of viewpoints. Biographical information on the Board members appears in Appendix B, along with a list of the members of each panel. PREPARATION AND ORGANIZATION OF THIS REPORT The current report is the sixth biennial report of ARLTAB. Its first biennial report was issued in 2000; annual reviews by the Board had been issued in 1996, 1997, and 1998. As with the earlier reviews, this report contains the Board’s judgments about the quality of ARL’s work (Chapters 2 through 7 focus on the individual directorates, and Chapter 8 provides a crosscutting overview of ARL). The rest of this chapter explains the rich set of interactions that support those judgments. The amount of information that is funneled to the Board, including the consensus evaluations of the recognized experts who make up the Board’s panels, provides a solid foundation for a thorough peer review. This review is based on a large amount of information received from ARL and on panel interac - tions with ARL staff. Most of the information exchange occurs during the annual meetings convened by the respective panels at the appropriate ARL sites. Both at scheduled meetings and in less formal interactions, ARL evinces a very healthy level of information exchange and acceptance of external com - ments. The assessment panels engaged in many constructive interactions with ARL staff during their annual site visits in 2009 and 2010. The dates of the panel site visits are included in the introductory section of Chapters 2 through 7 on the individual directorates. In addition, useful collegial exchanges took place between panel members and individual ARL investigators outside of scheduled meetings as ARL staff members sought additional clarification about panel comments or questions and drew on panel members’ contacts and sources of information. Each panel meeting lasted 2½ days, during which time the panel members received a combination of overview briefings by ARL management and technical briefings by ARL staff. Prior to the meetings, panels received extensive materials for review, including selected staff publications. The overview briefings brought the panels up to date on ARL’s long-range planning. This context- building step is needed because the panels are purposely composed mostly of people who—while experts does not have a panel specifically devoted to the Army Research Office, which is another unit of ARL, but all Board panels examine how well the in-house research and development of ARO and ARL are coordinated. Appendix A provides information summarizing the organization and resources of ARL and its directorates.

OCR for page 12
14 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY in the technical fields covered by the directorates(s) that they review—are not engaged in work focused on Army matters. Technical briefings for the panels focused on the R&D goals, strategies, methodolo - gies, and results of selected projects at the laboratory. Briefings were targeted toward coverage of a representative sample of each directorate’s work over the 2-year assessment cycle. Briefings included poster sessions that allowed direct interaction of the panelists with staff of other projects that either were not covered in the briefings or had been covered in prior years. Ample time during both overview and technical briefings was devoted to discussion, both to clarify the relevant panel’s understanding and to convey the immediate observations and understandings of individual panel members to ARL’s scientists and engineers. The panels also devoted sufficient time to closed-session deliberations, during which they developed consensus findings and identified important questions or gaps in panel understanding. Those questions or gaps were discussed during follow-up sessions with ARL staff so that the panel was confident of the accuracy and completeness of its assess - ments. Panel members continued to refine their findings, conclusions, and recommendations during written exchanges and teleconferences among themselves after the meetings. In addition to the insights that they gained from the panel meetings, Board members received exposure to ARL and its staff at Board meetings each winter. The 2010 Board meeting focused on the assessment process, and discussions with the ARL management team led to improvements in the assessment process in terms of the panels receiving timely read-ahead materials for their meetings, the development of the panel meeting agendas, and more attentiveness to the assessment process time lines. Also, some Board members attended the annual ARL Program Formulation Workshop in 2009 and 2010; at these workshops the ARL directorates discussed their programs with the directorates’ customers and stakeholders. ASSESSMENT CRITERIA During the assessment, the Board and its panels considered the following questions posed by the ARL Director: 1. Is the scientific quality of the research of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally? 2. Does the research program reflect a broad understanding of the underlying science and research conducted elsewhere? 3. Does the research employ the appropriate laboratory equipment and/or numerical models? 4. Are the qualifications of the research team compatible with the research challenge? 5. Are the facilities and laboratory equipment state of the art? 6. Does the research reflect an understanding of the Army’s requirement for the research or the analysis? 7. Are programs crafted to employ the appropriate mix of theory, computation, and experimentation? 8. Is the work sufficiently unique and appropriate to the ARL niche? 9. Are there especially promising projects that, with application of adequate resources, could produce outstanding results that could be transitioned ultimately to the field? Within the general framework described above, the Board also developed and the panels applied detailed assessment criteria organized in the following six categories (Appendix C presents the complete set of assessment criteria):

OCR for page 12
15 INTRODUCTION 1. Effectiveness of interaction with the scientific and technical community— criteria in this category relate to cognizance of and contributions to the scientific and technical community whose activi - ties are relevant to the work performed at ARL; 2. Impact on customers—criteria in this category relate to cognizance of and contributions in response to the needs of the Army customers who fund and benefit from ARL R&D; 3. Formulation of projects’ goals and plans—criteria in this category relate to the extent to which projects address ARL strategic goals and are planned effectively to achieve stated objectives; 4. R&D methodology—criteria in this category address the appropriateness of the hypotheses that drive the research, of the tools and methods applied to the collection and analysis of data, and of the judgments about future directions of the research; 5. Capabilities and resources—criteria in this category relate to whether current and projected equip- ment, facilities, and human resources are appropriate to achieve success of the projects; and 6. Responsiveness to the Board’s recommendations—with respect to this criterion, the Board does not consider itself to be an oversight committee. The Board has consistently found ARL to be extremely responsive to its advice, so the criterion of responsiveness encourages discussion of the variables and contextual factors that affect ARL’s implementation of responses to recommenda - tions rather than an accounting of responses to the Board’s recommendations. APPROACH TAKEN DURING THE REPORT PREPARATION This report represents the Board’s consensus findings and recommendations, developed through deliberations that included consideration of the notes prepared by the panel members summarizing their assessments. The Board’s aim with this report is to provide guidance to the ARL Director that will help ARL sustain its process of continuous improvement. To that end, the Board examined its extensive and detailed notes from the many Board, panel, and individual interactions with ARL over the 2009-2010 period. From those notes it distilled a shorter list of the main trends, opportunities, and challenges that merit attention at the level of the ARL Director. The Board used that list as the basis for this report. Specific ARL projects are used to illustrate these points in the following chapters when it is helpful to do so, but the Board did not aim to present the Director with a detailed account of 2 years’ worth of interactions with bench scientists. The draft of this report was subsequently honed and reviewed accord - ing to NRC procedures before being released. The approach to the assessment by the Board and its panels relied on the experience, technical knowledge, and expertise of its members, whose backgrounds were carefully matched to the techni - cal areas within which the ARL activities are conducted. The Board and its panels reviewed selected examples of the standards and measurements activities and the technological research presented by ARL; it was not possible to review all ARL programs and projects exhaustively. The Board’s goal was to identify and report salient examples of accomplishments and opportunities for further improvement with respect to the technical merit of the ARL work, its perceived relevance to ARL’s definition of its mission, and apparent specific elements of ARL’s resource infrastructure that is intended to support the technical work. Collectively, these highlighted examples for each ARL directorate are intended to por- tray an overall impression of the laboratory while preserving useful mention of suggestions specific to projects and programs that the Board considered to be of special note within the set of those examined. The Board applied a largely qualitative rather than quantitative approach to the assessment; it is possible that future assessments will be informed by further consideration of various analytical methods that can be applied. The assessment panels’ site visits are currently scheduled to be repeated annually and the assessment report to be issued biennially.

OCR for page 12
16 2009–2010 ASSESSMENT OF THE ARMY RESEARCH LABORATORY REPORT CONTENT This chapter discusses the biennial assessment process used by ARLTAB and its six panels. Chap - ters 2 through 7 provide a detailed assessment of each of the six ARL directorates. Chapter 8 presents an overview focused on crosscutting issues across all of ARL. The appendixes provide the ARL orga - nizational chart and staffing profile, biographical information on the members of ARLTAB and a list of the membership of its panels, the assessment criteria used by ARLTAB and its panels, and a list of acronyms found in the report.