Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 17
2009-2010 Assessment of the Army Research Laboratory 2 Computational and Information Sciences Directorate INTRODUCTION The Computational and Information Sciences Directorate (CISD) was reviewed as a whole by the Panel on Digitization and Communications Science during the periods July 7-9, 2009, and July 6-8, 2010. The reviews consisted of overviews given by management of the directorate and its divisions, presentations on a subset of current projects, poster sessions at which project leaders were available, and laboratory tours. CISD has stabilized its organization at the same four research divisions discussed in the previous report of the Army Research Laboratory Technical Assessment Board (ARLTAB)1: the Advanced Computing and Computational Sciences Division (AC&CSD), Battlefield Environment Division (BED), Information Sciences Division (ISD), and Network Sciences Division (NSD). CISD is responsible for Collaborative Technology Alliances (CTAs) on Communications and Networking (this CTA ended in FY 2009) and on Networks (started in FY 2010), a continuing International Technology Alliance (ITA) on Network and Information Sciences, and a Mobile Network Modeling Institute. CISD is also responsible for the Army High Performance Computing Research Center at Stanford University. CISD’s expressed mission is unchanged from that cited in the previous ARLTAB report: to create, exploit, and harvest innovative technologies to enable knowledge superiority for the warfighter through advanced computing, network and communication sciences, information assurance, and battlespace environments. To carry out this mission, CISD performs research for the following purposes: 1 National Research Council. 2009. 2007-2008 Assessment of the Army Research Laboratory. Washington, D.C.: The National Academies Press.
OCR for page 18
2009-2010 Assessment of the Army Research Laboratory To advance computational sciences and high-performance computing (HPC) technologies in support of Army systems; To enhance warfighter effectiveness through environmental knowledge and technology; To provide fused, timely information from all relevant sources to the warfighter; and To develop self-configuring wireless network technologies that enable secure, scalable, energy-efficient, and survivable tactical networks. ARLTAB’s previous report highlighted continuing significant advances made by CISD in the machine translation (MT) of foreign languages, atmospheric acoustics and radio-frequency (RF) and optical propagation in battlefield environments, and surface-level weather modeling. Promising advances were reported in experimental sensor systems (chemical and particle detection in aerosols and the atmosphere, microfluidic devices, and quantum dot formation for night-vision goggles); in the transition of previous prototype systems to products (e.g., achieving a 100-times reduction in the weight of a compact lidar system); in the development of a small, standardized battlefield network interface called the Blue Radio; and in theory and modeling (improving models of turbulence in the atmosphere, codes to design application-specific microfluidic devices, and calculations of binding affinities for potentially toxic chemical and biological compounds). All of these advances had two common characteristics: solid science and a clear understanding of the relationship to real Army problems. Areas that were deemed challenges in the previous ARLTAB report included a need for more of a systems engineering outlook in projects, validation and verification (V&V) of models and computer codes, an increasing need to perform sophisticated analyses and data mining on experimental data, and the need to find a way to leverage the work done to date on Blue Radio (especially how to achieve a single-chip implementation that could be used in a variety of systems). The challenge of expanding beyond traditional computing applications could benefit, as indicated in the previous report, from the development of an ability to accumulate, analyze, understand, and efficiently process human and electronic intelligence about relationships between individuals and organizations in an asymmetric battlespace. Challenges also existed in moving the weather-modeling efforts from a concentration on the modeling of atmospheric physics to the building of real applications on top of such models. In the high-performance computing area, a need for additional research and development (R&D) resources was identified, specifically for developing a professional staff that is capable of building HPC software products that are efficient and application-specific. There also appeared to be a lack of HPC vision in the non-AC&CSD divisions of CISD. Table A.1 in Appendix A characterizes the staffing profile for CISD. CHANGES SINCE THE PREVIOUS REVIEW Since the site visit of the panel in 2009, there has been no change in the overall organization of CISD and only one management change at the level of division chief. This degree of stability is in positive sharp contrast to the situation in prior years. In terms of Collaborative Technology Alliances, both the Communications and Networking CTA and the Advanced Decision Architectures CTA (which CISD partnered with the Human Research and Engineering Directorate [HRED]) were completed, and work continued on the International Technology Alliance on Network and Information Sciences, initiated in 2008. The focus of this ITA is on managing end-to-end information flows in support of coalition decision making. The Networks CTA was started in FY 2010.
OCR for page 19
2009-2010 Assessment of the Army Research Laboratory ACCOMPLISHMENTS AND ADVANCEMENTS Document Management for Foreign-Language Machine Translation CISD continues to demonstrate extraordinary advances in deployable facilities for the machine translation of foreign-language material in ways that are clearly relevant to real Army problems. The focus of prior related research has been first on speech translation and then on text translation, with an emphasis on the rigorous evaluation of alternative algorithms and on combining them to improve translation performance. In this reporting period, CISD reported a program of research that did not address machine translation itself, nor the merits of various algorithms and software; instead, it addressed putting machine translation into a key downrange application. Given that the Army is composed almost entirely of people whose native language is English and that most of its overseas deployments are in non-English-speaking parts of the world, developing tools to help English-speaking soldiers leverage MT technology is a very worthy research direction for ARL. In particular, the research reviewed in this period focused on creating and optimizing work flows for leveraging existing MT technology to address the automated analysis of large volumes of foreign-language paper documents. The objective is a cradle-to-grave process that starts with scanning and optical character recognition (OCR), continues through machine translation, and ends with the categorization, analysis, and storage (in a searchable and retrievable fashion) of the content—in English. The selected work flow is adaptive, and it can be arranged to process documents automatically or semiautomatically (with human annotations), or customized to use low-level domain information. Particularly impressive is that readers who do not read Arabic and are not linguists could be part of the flow for unclassifiable documents, recognizing the structure of a document from nontextual clues (format, letterheads, pictures) and inputting this information to aid the MT process. The evaluation process for this work continued the solid approaches employed in earlier years and was done by using the Anfal corpus, produced after the First Gulf War, which is very large and contains many different forms of documentation and correspondence. Particularly notable was the use of well-conceived statistical techniques to bin the samples and organize the work. Similar work addressed the translation of domain-specific texts from English to Dari, something that is of real value in the Afghanistan theater. This work also is a powerful example of the great potential for exploiting synergies across multiple projects toward sustainable innovations that could move forward the data-to-decision mission of CISD in general. In particular, with a focus on interoperable data acquisition, processing, and metadata services, there is the potential to develop rich data collections that could not be developed otherwise. Additionally, methods for data exploitation share many common algorithms related to clustering, classification, and network analysis that could benefit from shared approaches to scaling to bigger data sets through performance modeling and parallel computing. New optimization methods or rule-based systems could also be developed for translating these algorithmic outputs to new, Army-relevant decision-support measures and metrics that can be adapted and tuned through experiment and feedback from end users. It is therefore important to provide the needed resources and support for a systematic specification and development process with a data sets approach to code reuse and interoperability in order to enable the development of new sustainable pathways for growing this emerging area. Quantum Ghost Imaging Developing imaging sensors for bad weather environments has long been an Army Research Laboratory (ARL) strength, bolstered by world-class work on infrared (IR) sensors at the Sensors and Electron
OCR for page 20
2009-2010 Assessment of the Army Research Laboratory Devices Directorate (SEDD). In this reporting period, however, new work has combined ARL’s strengths in quantum physics with advanced high-performance computing to perform first-in-the-world demonstrations for what can be described as a potentially entirely new way of imaging through scattering and absorbing media. Light from a scene is split into a conventional charge-coupled device camera and a single-pixel sensor, with a point-to-point 2-photon Glauber coherence correlation, to form an image. Performing this correlation is a very heavily computationally-intensive problem. The significance of the work was clear both from the demonstrations provided for the panel and from external validation through both multiple patents and multiple publications in very-high-impact publications with rigorous peer review. This is outstanding work in all respects—based on truly fundamental and novel physical design—and it is likely to lead to high-impact relevance for Army (and other) applications. It is an excellent example of a project based on fundamental physical drivers and real Army applications. It is clear that there are multiple paths to both deploying and enhancing the technology and that such work is deserving of continued ARL support. Collaborations with HPC expertise should lead to the ability to perform the computations in closer to real time in a sensor platform compatible with battlefield conditions or conditions familiar to civilian first-responders (firefighters). Additional computation and/or sensing may also lead to ranging determination and/or increased resolution. Social Network Construction from Unstructured Input, Information Fusion, and Analysis Over the past few years, the panel has encouraged CISD to explore computational tools to aid in performing social network analysis (SNA) based on intelligence and other data sources; in this sense a “social network” represents the relationships between humans. In the Army context, the most obvious but not the only example is the identification of the combatants engaged across the improvised explosive devices (IEDs) chain and the mapping of their functions, from the financiers, through the bomb makers and parts suppliers, to the IED placement and detonation teams. Results in Iraq using anthropologists, intelligence personnel, and relatively ad hoc tools have proven the potential value in trying to automate even further the construction of such human networks from disparate data sources. CISD had acted on a previous panel’s suggestion, hiring a social scientist to collaborate with a group of CISD computer scientists. Commendably, that social scientist is actually leading CISD colleagues in automating the fusion of data sources and the generation of the metadata tags on such data, which then allow for the kind of graph analysis that can reveal adversary networks. The discussion above on machine translation and analysis of documents provides an example of such data sources. The SNA thrust being performed focuses on constructing and analyzing social networks from sparsely tagged, unstructured data for tactical data-to-decision relationship discovery service. The work is still early in its execution, but the hiring of new Ph.D.’s and the funding of Small Business Innovation Research (SBIR) projects indicate a seriousness that should have positive long-term consequences. The approach includes acquiring Army-relevant soft and hard data, then processing the data to derive metadata, followed by analysis on the SNA structure to reveal microscale relations and by analysis on the dynamics of network change with respect to macroscale events or processes that evolve on network structures over time. The SNA team also appears to understand and articulate the key challenges in the task of integrating soft and hard data; the team was open to exploring the potential of new data-collection modalities that may help remove some of the underlying dichotomies, while at the same time working with another team on the integration issues. Of particular note is the deliberate choice to do early experiments on terabyte data sets in order to assess the challenging aspects of the problem in scales much larger than those of toy
OCR for page 21
2009-2010 Assessment of the Army Research Laboratory data sets in ways that are critical for meeting the discovery needs and testing of new analysis algorithms. The clear articulation of project goals and plans and the project’s relevance to the Army mission were impressive. Additional strengths concerned the support of three new SBIR projects on source selection with strong potential for near-term successes; a focus on interdisciplinary collaboration; expanded interactions with the Office of the Secretary of Defense, the Multidisciplinary University Research Initiative, CTA, and end users; and initial experiments with Twitter feeds aiming at the development of a system that can incorporate streaming data. Related work that also showed good experimental design and potential high relevance to Army needs was a study of how better visualization techniques (i.e., how to cluster for display purposes large amounts of disparate data) aid in reducing the timing of decision making in high-tempo workloads. The experiments were conducted with Reserve Officers’ Training Corps (ROTC) students as subjects, with Army-relevant challenges. Collaboration with both academic institutions and HRED was excellent, with a strong transition plan to two Army Technology Objectives (ATOs): Advanced All-Source Fusion (A2SF) and Tactical Human Integration of Networked Knowledge (THINK). Further work in several other areas might improve even more the future value of the overall SNA work, including the role of statistical analysis to quantify uncertainty, a more formal look at the algorithmic aspects of the underlying SNA schemes, and the role of different disciplines, including physics, biology, and computer science, in developing a team with the right set of skills and expertise. Plans for effective development and deployment of the relationship discovery service could benefit from bringing underlying systems and software engineering challenges to the forefront. The interdisciplinary team approach to meeting project requirements is commendable; CISD should consider identifying an additional systems leader—for example, one trained in computer science research with some explicit cross-training in the appropriate social sciences. High-Performance Computing While managing some world-class computing facilities, the Advanced Computing and Computational Sciences Division has perhaps the least resources for research in all of CISD, in terms of both dollars and Ph.D.-level researchers. Given this limitation, in its previous report ARLTAB recommended that such resources as were available be focused on encouraging and developing an HPC capability in a much broader fashion than currently exists throughout ARL, and on developing software that ARL can take advantage of in future petaflops systems. In this 2009-2010 period, AC&CSD has made some significant strides in doing so. In terms of producing HPC applications, the division has focused on developing a research portfolio that emphasizes applications that clearly cross divisional boundaries, such as support for lightweight combat systems and computational nanoscience and biological science. Although some of the work still suffers from a lack of focus on V&V of both the code and the underlying models, other work—such as modeling materials with complex microstructures that are hit with a shock of some kind, and the modeling of antimicrobial peptides with bacteria membranes—seems to be making real strides, particularly since the researchers focused on a good formulation of the problem with a solid initial approach, developed helpful collaborations with leading academic centers, and for at least early work included sound checks on the potential limits of the models (e.g., validity of a model when applied to more complex environments or processes). CISD should consider how to integrate such applications with other tools in a productive fashion, and it should also consider how to ensure a deployment path so that, if the effort is successful, the resulting code is not dropped, as was done with a prior Object Oriented MicroMagnetic Framework package.
OCR for page 22
2009-2010 Assessment of the Army Research Laboratory In terms of addressing the rise of petascale computing, new efforts are underway to begin considering nontraditional HPC, namely, the potential in the near future of moving today’s petascale supercomputing systems down into forms relevant to tactical environments, such as the back of a Humvee. Included are efforts to investigate new hardware technologies such as field-programmable gate arrays (FPGAs) and graphics processing units for computing-intensive applications such as tactical radars (synthetic aperture radars for detecting IEDs), battlefield weather prediction, RF propagation models, and mission planning, and also to investigate these technologies on domain-specific programming environments and compilers that might leverage such hardware in a more productive fashion than could be done using traditional tools. Neither these platforms nor the applications are traditional HPC targets, so it is not unexpected that there is a learning curve, with some false starts. However, the directions taken by CISD are very reasonable ones that should over the long term result in real new capabilities. Battlefield Weather The efforts of the Battlefield Environment Division are clearly focused squarely on the key problems identified in the white paper entitled Army Weather Support from the Army Intelligence Center, with continued high levels of demonstrated expertise in virtually all areas that the division has addressed. In what might be called traditional weather prediction, BED has provided repeated instances showing significant and steady progress toward the ability to develop very short term predictions for small locales (battlefields). Improvement is needed with respect to defining a basis for comparisons between modeled and measured data—that is, there is need to define a better metric for what is good enough for Army applications. BED has seized a clear leadership position in the examination of effects of atmospheric turbulence, especially at or near the surface of Earth and in urban environments. A fundamental approach to examining these effects is key for understanding sound propagation in the battlefield (e.g., shooter location), updated accuracy of long-range artillery, effects on small robotic flyers (especially near buildings), the mixing of aerosol particles (smoke, fog, hazardous particles) into the atmosphere, and effects on optical signals. BED has demonstrated year after year continued advances in all of these areas, although there have been some concerns about nuances of the physics being modeled. BED has, however, leveraged a long series of experiments, mostly at its White Sands, New Mexico, facility, which are largely unique and provide a solid basis for validating the models. Related to this work is a series of projects addressing atmospheric effects on nontraditional imaging techniques, particularly those involving terahertz radiation sources and using polarimetric signatures. This research has been making continued and good progress over the past couple of years, but it has been to some extent paced by the available hardware. The quality of the data gathering and the well-defined motivation are particularly important. The techniques are not ready for system deployment yet but are moving in that direction. BED should continue efforts to propose possible operational concepts to leverage this work, particularly as advances are made in AC&CSD to develop tactical HPC systems with the computational power needed for these applications. OPPORTUNITIES AND CHALLENGES Potential Crosscutting Issues Prior ARLTAB reports have suggested several technology issues that have cut across multiple ARL directorates, but with particular impact on CISD. Examples include advanced computing, networking
OCR for page 23
2009-2010 Assessment of the Army Research Laboratory (especially ad hoc), information fusion, information security, system-of-systems analysis (SoSA), prototyping, and verification and validation. The current report notes the commendable start of efforts that address several of these issues (advanced computing, information fusion, networking). Others, such as information security, SoSA, and V&V, remain issues that could benefit from more crosscutting activities. In addition, however, several new areas surfaced in the 2009-2010 reporting period that may also qualify for ARL-wide consideration: Microrobotics: The need for surveillance, especially at the squad level, has continued to explode, and the introduction of smaller and smaller platforms is continuing to offer new opportunities for deployment. CISD is potentially at the heart of such systems, which involve networking, information fusion, and high-performance onboard image processing—obvious candidates that would engage ISD, NSD, and AS&CSD, but the ability to carry weather detectors and/or to be influenced by micro weather events clearly is also relevant to BED. New capabilities such as swarming and electronic warfare (jamming) will also clearly involve not only CISD but will also interact with SEDD in the development of new sensors compatible with limited resources, and with the Vehicle Technology Directorate (VTD) when computing can simplify platforms. Power: Energy consumption has become a first-class design constraint on almost all Army platforms, especially as more and more functionality is done with computing. Prognostics and diagnostics (platform-based fault detection and reconfiguration): The increase in platform complexity and the increasingly rapid schedule of introduction, deployment, and retirement mean that the average soldier who must deal with complex equipment barely has time to learn to use a new system, let alone to become expert enough to be able to repair or reconfigure it. Computing must take a central role in the automation of platform-based fault detection and reconfiguration, but it must do so in ways that are compatible with the platforms and that simplify the soldier’s overall workload. CISD needs to be involved both in platform-based fault detection and reconfiguration and in remote real-time data mining, parameter extraction, trend analysis, and real-time modeling. Biomechanics: Although CISD does not have as central a role in biomechanics as exists in HRED, there certainly will be a need to develop and then support significant modeling activities, particularly using HPC expertise, facilities, and resources. Acoustics: Already a strong area in CISD’s research portfolio, acoustics will increase in importance as additional sensors and additional laboratories such as HRED’s Environment for Auditory Research (EAR) come online and require modeling support, data visualization, and correlation with atmospheric effects. Modeling and computational science: This area clearly overlaps multiple components of CISD’s charter, and it remains, through several ARLTAB reviews, an identified area for crosscutting activities. The issue of identifying potentially disruptive technologies that might radically change the problems which face the Army (such as the rise of asymmetrical warfare and IEDs) and the way in which the Army needs to leverage technology to respond to them are of ARL-wide importance. However, in the very fast moving technologies that are the realm of CISD, change—presenting both threats and opportunities—probably occurs faster than in other directorates. Thus each of CISD’s divisions, and CISD as a whole, may benefit from an explicit recognition of the potential of such technologies and the development of a formal mechanism to help identify critical technologies in a timely fashion.
OCR for page 24
2009-2010 Assessment of the Army Research Laboratory Challenges in Networking The previous report of ARLTAB commented favorably on the potential created by the formation of the Network Sciences Division and the related Mobile Network Modeling Institute. At the time of that assessment, ARL indicated that this development was in part an outgrowth of prior ARLTAB assessment findings which stated that a variety of issues associated with mobile networks had risen to be of crosscutting importance to ARL. The issues that ARLTAB had discussed as being important to the Army included network security, ad hoc wireless networks in particular, system prototyping, and model validation and verification. The structure of the then-new NSD focused on three levels of the problem: tactical network assurance, networking sciences development, and sustaining base network assurance. This structure continues to represent an R&D capability that, if developed appropriately, could have significant impact on a wide range of real current and future Army problems. One example of an R&D achievement in this area is an attempt to use small, covert perturbations of a signal between two wireless nodes to increase the probability of identifying a valid node and likewise to increase the probability of identifying a node introduced into a network for nefarious purposes. This work won Best Army Conference Paper of the Year and Invention of the Year awards. However, there are still challenges to the fulfillment of these capabilities in operational systems. Networking is still a new area, but it would help both the evaluation and the applicability of the work to have a more careful articulation of the key problems being attacked and of how the research is being directed to address them. For example, the work reviewed in this cycle tended to be divided between strong theory, with the relationship to real Army problems not being crisply described, and work that had a good Army connection but that had lost the connection with networking. The former, theory-related work (on multi-hopping in cognitive networks and statistical interference) was usually good but not necessarily cutting edge, and it often made early assumptions with little perceivable justification with respect to how the work might relate to real applications, especially with relevance for the Army. The latter work (such as that on distributed information quality or component-based routing) had good Army motivation, but it seemed to define networks in a much more abstract sense than that used to establish NSD or the Mobile Network Modeling Institute, and it often seemed to be more closely related to Information Sciences Division problems than to networking issues. This latter work also lacked something that is fairly typical for classical network problems, namely, metrics of correctness and success that can be measured or estimated in a meaningful fashion. Related continuing concerns, discussed below, about the validation and verification of network models, are along the same lines as those articulated in many previous ARLTAB reports. Standing the Mobile Network Modeling Institute had as one of its goals the creation of a capability to simulate, emulate, and model networks in ways that could shorten the time to do design exploration, prototyping, and deployment. Such models include ones that cover protocols, waveform propagation in the environment, and traffic models. However, there is still not enough effort being made to ensure that these models, especially when run in a multiscale, multilevel, end-to-end mode, are in fact reliable predictors of reality. The suggestion from previous reports continues: CISD should perform joint experimental and modeling efforts using devices such as the Blue Radio to achieve such V&V. Related to the preceding issue is a concern that there may not be as full an understanding of the state of the art as is needed to address the leading-edge problems facing ARL and to avoid activities that are not as calibrated to the state of the art elsewhere as they might be. CISD should consider continuing to send some of its Ph.D. researchers to top venues and then have them organize ARL workshops at which data sets developed in NSD and the institute can be used to help promulgate the problems on which ARL is focused and to help evaluate whether answers from external research are relevant (much
OCR for page 25
2009-2010 Assessment of the Army Research Laboratory as has been done over the years by the CISD machine translation group). Another suggestion would be that CISD strengthen ties with some leading external research groups such as the Research Laboratory of Electronics at the Massachusetts Institute of Technology. There is also potential in NSD to implement a practice that is currently employed in BED—namely, the accumulation of unique, world-class data sets from experimental and/or modeling efforts and the documenting of these data sets in ways that will allow them to be useful both to ARL in its future efforts and to the larger R&D community. Challenges in Battlefield Environment Studies The research portfolio of the BED has for years been an exemplar of a good balance between modeling and experimental efforts—especially, for example, in testing in realistic theaters such as at the world-class facilities at White Sands, New Mexico, and in multidisciplinary efforts such as the use of HPC. For the most part this continues, but with some challenges as new areas (such as aerosol dispersion or ultrasonics) are entered and more complex environments (such as urban areas, with turbulence) are attempted. These challenges lie largely in the articulation by researchers of their understanding of what it takes to validate a new model, and with mapping out in the long term how the results of such research might be deployed. There is, however, an excellent roadmap in place for traditional weather-related work, and BED should consider fleshing out in that roadmap possible alternatives for these new areas. Challenges in Decision-Aiding Systems The work discussed above about social network analysis clearly is key to fusing information and aiding the warfighter in making decisions. There were, however, still challenges in elevating and applying research in the area. In the current review cycle, several projects exhibited such challenges. First, related to the problem of managing and searching huge knowledge bases were some initial efforts, reported in 2010, at using a new software paradigm called Hadoop (an implementation of the MapReduce programming model) to solve the entity resolution problem—the problem of disambiguating a name from many possible matches. Hadoop was started by Google, Inc., to attack the same type of massive data-intensive applications that are liable to occur in SNA problems. The problem chosen for study is an embarrassingly parallel problem well suited to this approach. In addition, Hadoop has been adopted by the Army Intelligence Security Command (INSCOM), and so using this approach will simplify integration into potential future customer systems. The work was a reasonable initial project for learning about the Hadoop technology, but it was not about research into methods for entity resolution; rather, it was an evaluation of some existing tools for the entity resolution problem, making use of two public databases containing the names of movie actors. As appropriate as the approach may be, the current project had some significant challenges. As an evaluation of existing software, the overall problem was not well formulated. For example, how would results on these public databases translate to databases of interest to INSCOM? The latter are likely to be much larger and may also contain more errors or have more complex entities (indeed, a separate CISD presentation explained that, in Iraq, identifying a particular individual may require both the person’s name and where that person sells in the marketplace). Without a simple analytic performance model to reflect a scaling of the database size, the initial experiments cannot be placed in context and can only suggest, not predict, the performance against larger data sets. The work was not sufficiently connected to the tactical needs of the Army or to other, related work in the broader intelligence community, and the project needs to put its work in that context. The project needs to have a clearly stated objective that is
OCR for page 26
2009-2010 Assessment of the Army Research Laboratory consistent with the mission of ARL. This is important technology—CISD should develop a clearer set of objectives and an experimental design that addresses those objectives, along with closer interaction with both INSCOM and the broader intelligence community. Another example with respect to challenges in decision-aiding systems was an attempt to characterize team decision making (i.e., a human network) experimentally as a function of information loss and delay due to data network issues; an existing platform called ELICIT was used to represent information sources, servers, and decision makers, and validation of agent-based experiments was done through human-in-the-loop experiments. The fundamental concept behind this work was sound—that is, determining the impact of variations in network quality of service on decision making is an important Army need. However, without care in performing such human-computer network experiments, the actual value to the Army could be limited. In particular, using just delay and loss measures in the data network models seem limiting, especially in terms of parameters that are relevant to how humans interact with such networks. Better coordination with HRED and the Communications-Electronics Research, Development, and Engineering Center may be able to help with realistic assumptions. However, the real problem today is collaborative decision making (the human side of the networks), but it was not obvious how the results would relate to parameters such as correctness of decision or how the results would be validated other than by comparison. CISD should consider introducing U.S. Army Training and Doctrine Command-validated behaviors into research and interacting with the Joint Experimentation Directorate. Challenges in Robotic Autonomy A growing asset for soldiers is the use of small, robotic platforms to go where there may be significant unknown danger. Key parts of making such platforms usable, especially in complex urban terrains, are communication between the platform and the troops and navigation of the platform in ways that permit a wide range of semiautonomous behaviors. Over the past few years, CISD has presented to the panel the results of a series of reports on related projects from multiple divisions within CISD. Given the criticality of the problems and the potential value added by real solutions, the continued effort is commendable. However, there are challenges, primarily in developing system concepts that are realistic and might be fieldable. (Both of these points are examples of the potential usefulness of a more consistent systems engineering focus, as discussed below.) One example of such challenges is a project looking at the deployment of robots to form a dynamic network where various platforms may be out of the line of sight of the troops (e.g., inside a building on the other side of corridors). The problem posed was clearly a difficult one, but it suffered from an assumption that a map existed at the beginning and from questions about radio-frequency propagation and map construction, especially in the three-dimensional environment represented by a building. Also, one would expect that topologies that represent richer than a minimum connectivity would be more valuable in a warfighting environment where intermediate relays may be destroyed or go off-line. A related project involves indoor navigation in a Global Positioning System-denied environment using a combination of out-of-building sensors and dead reckoning based on a simple inertial navigation unit tied to a soldier’s foot. The project’s first year of work was reported to have been focused on the capabilities of a simple inertial measurement unit (IMU). The panel thought that the challenge was not as much in the actual work or approach (both seem to be of high quality) as in operational concepts issues: What happens when a soldier is not walking but crawling? How are initial locations established? How can external sensors help? Many of these concerns could be mitigated by some coordination with combat forces personnel.
OCR for page 27
2009-2010 Assessment of the Army Research Laboratory Another related project was one whose goal is an algorithm to estimate a robot’s position and movement by combining epipolar lines from images with IMU data, with demonstration on PackBots. This is an excellent problem, with direct application to a number of Army problems. However, there is a potential challenge related to how this research might transition to a relevant deployable system: What aspects of IMU-driven navigation would be improved (and by how much)? What kind of processing needs to be onboard the platform to be usable in a real-time setting? What are the lower-level details of the filters assumed? Challenges in System Engineering As noted in prior ARLTAB reports, it remains a significant challenge across CISD to ensure that even in relatively basic research programs a good understanding is formed about how potential systems that might be developed out of such research might be deployed and used in real Army scenarios. A small amount of systems engineering early in many programs could help avoid paths to systems that, even if successful, would be difficult to deploy in real-time systems or would have some obvious characteristic that would make them impractical. And, conversely, a small amount of systems engineering early in many programs could provide insight into alternatives that would mesh much better with practice. The same systems engineering focus would also help enable early comparison of research goals with expected roadmaps for established technologies and help prepare realistic statements of the potential gains from the new technologies being researched. It is insufficient to pursue the development of technologies that are better than those of today, but which may be potentially only on a par with what is expected from the natural progression of technology. Challenges in Evaluation, Validation and Verification, and Experiment Design Another general comment from prior ARLTAB reports that remains a challenge across CISD involves testing and evaluation for experimentally driven programs and the validation and verification of models—validating that models developed during research programs actually reflect reality and verifying that the codes or systems that are constructed are in fact correct implementations of the models. There are research areas such as machine translation in which a V&V mind-set has become central to the research process, and others with apparently little such focus. In still others, particularly projects involving complex computations, there continues to be a tendency to develop stand-alone codes, without any clear approach articulated as to how to ensure that both the algorithm modeling the physics and the implementation of that algorithm are correct. A tendency to believe the machine is evident, and it needs to be avoided by formal verification. A key example of a systematic need for V&V is in the networking area, as discussed above, especially in the modeling area. The signal-propagation models used for such systems as soldier-mounted mobile networks or sensor networks must match the actual close-to-the-ground physics of the battlefield, while assuming the kinds of transmission waveforms that are liable to be used. The traffic patterns used to drive studies of inherently low power protocols must reflect reasonable configurations, particularly in mobile cases, or the results are potentially misleading. If done right, solving the V&V dilemma can, as a side effect, provide significant long-term benefit for future research, both in ARL and in the larger community. BED has for a long time taken care to save the data sets that come out of V&V experiments, and that has provided a rich knowledge base to fuel future work. A formal process for doing the same in other areas, such as networking, will possibly have similar long-term value.
OCR for page 28
2009-2010 Assessment of the Army Research Laboratory The previous ARLTAB report also discussed another emerging general need: that of performing sophisticated analyses on experimental data. Such analyses involve both classical statistical computation and, perhaps more importantly, information extraction from large and often unstructured data sets. Data mining has emerged in the commercial world as key for applications ranging from determining personalized online purchase preferences to performing portfolio analyses. In this 2009-2010 review cycle, CISD described several projects to begin such efforts, including work on Hadoop—the technique that has grown out of the Web search and services world to use large numbers of computers in an organized fashion to perform very fast unstructured searches on queries that are generated on the fly. This work is a valuable learning experience, but it is not research yet. Questions on what kind of data sets are relevant and, more importantly, on how their size must scale, at the tactical versus operational versus strategic level, were not articulated, and so questions such as where the technology works or breaks are largely unanswered. (This is also an example of the need for a systems engineering perspective, as discussed earlier.) A challenge related to those discussed above, but somewhat different from those articulated in the past, became evident during this review cycle. More and more research projects are faced with multidimensional design spaces with large numbers of possible parameter values, leading to combinatorial explosions in the number of cases to be explored. In multiple research projects observed by ARLTAB, CISD often applied ad hoc approaches to determine which subsets of these design spaces would be explored and how the results from one set of explorations would be used in the next set. This was true for both model-driven work and experiment-driven work. Some in-house expertise in experiment design could prove invaluable both in reducing the resources needed for the explorations and in raising confidence that more near-optimal solution points can be found. Challenges in Work Flow Analysis Prior sections of this chapter comment favorably on the high quality of the work on developing a process for transforming large numbers of paper documents in non-English languages into information stores of real use to analysts. Other research within CISD also has a similar focus on automating what today are people-intensive processes. Independent of the usually high quality of the technical work involved in the automation process itself, there seems to be a need to better define and track metrics that adequately reflect the reduction in the human workload and/or the extension of human experts—productivity that results from employment of the automated work flow. The general question of how errors propagate through the system could benefit from more thought, as could an articulation of plans for completing the research and transitioning the technology. The Defense Advanced Research Projects Agency, it should be noted, is not a transition target. Many of these issues are of the kind that a systems engineering viewpoint would help, in this case as possibly found in HRED. OVERALL TECHNICAL QUALITY OF THE WORK For this 2009-2010 review, the Board was asked to comment on several criteria. The first asks if the scientific quality of the research is of comparable technical quality to that executed in leading federal, university, and/or industrial laboratories both nationally and internationally. As in prior years, the answer is generally affirmative for CISD, which has exceptional expertise in selected areas such as weather, the use of quantum effects for fundamentally new imaging systems, and the leveraging of machine translation technology. Collaboration with external research entities, especially universities, again continues to be widespread throughout the directorate, although concerns continue about how much of the research that is reported had been done by or transferred to ARL, and to CISD in particular.
OCR for page 29
2009-2010 Assessment of the Army Research Laboratory Although the scientific and engineering staff are, on the whole, conducting and publishing quality research in a number of areas, a continuing concern is that there does not seem to be much involvement by staff members in leading scientific societies and organizations. Promoting such involvement should give rise to more scientific recognition and stature for the research staff, make them more aware of the state of the art in other groups, and make ARL more attractive to new Ph.D.’s. In addition to encouraging participation on review panels and editorial boards, consideration should be given to encouraging researchers, especially newer members of the staff, to help in organizing and hosting workshops on issues of relevance to ARL. Where possible, access might be provided to repositories of data sets that represent experiments of relevance to the topics of such workshops. The second criterion on which the Board was asked to comment questions whether the research program reflects a broad understanding of the underlying science and research conducted elsewhere. The answer here is essentially the same as in the previous ARLTAB report: the conclusions for various projects are mixed, with the areas mentioned above being exceptional. Success is especially evident for areas that have emphasized testing and evaluation, such as weather and machine-based language translation. However, in other areas such as data mining, where there is neither a history of prior internal projects nor collaborations in that area with others outside ARL, there is a distinct drop-off in an understanding of other work or the availability of existing program packages. In terms of the third criterion—whether facilities and laboratory equipment are state of the art—the answer is again the same as in prior years: a largely solid “Yes.” In many cases, it is not necessarily the equipment but the planning for using that equipment more effectively that could benefit from additional attention. There is a continuing concern about using the appropriate numerical models, especially within the Mobile Network Modeling Institute. The fourth criterion addresses the qualifications of the research team versus the research challenges. With just a few exceptions, the match seems to be adequate. In addition, the aggressive hiring of significant numbers of new Ph.D.’s in all divisions and the continued encouragement of Ph.D.-level work by current employees are a very positive indication. The hiring of a social scientist to lead a major CISD research group is an indication that ARL understands the need to grow expertise that is truly multidisciplinary. The success in hiring, given the long timescales that seem to be enforced on the hiring process by the bureaucracy, is remarkable. Creatively using postdoctoral research positions to evaluate new Ph.D.’s and giving them insight into the kinds of opportunities available at ARL should be continued. CISD research generally reflects an understanding of the Army’s requirements, although this focus is sometimes lost on projects of a more theoretical bent, such as some in the networking area, when a project needs to attempt transition from project formulation to the selection of data set characteristics that should be representative of Army scenarios. The next criterion deals with the structure of programs in terms of employing the appropriate mix of theory, computation, and experimentation. The results here are again mixed. In cases where projects effectively take advantage of ARL’s outstanding test facilities and weave in a feedback path that validates theory and drives more robust algorithm and system development, the results are usually strong, with obvious opportunities for transition. In other cases, where the use of facilities or the feedback of validation results is lacking or weak, effectiveness appeared less than optimal. The long-running issues related to V&V still remain, with the emergence of intelligent experiment design during this review cycle as something that, if improved, might enhance both the quality of results and the efficiency of achieving them. As indicated in prior assessment reports, the CISD management and research teams remain responsive to ARLTAB’s recommendations. CISD has instituted significant organizational changes, especially in the networking and HPC areas, that seem to be directly focused on alleviating problems that ARLTAB
OCR for page 30
2009-2010 Assessment of the Army Research Laboratory had commented on in the past. The NSD and associated institutes and other initiatives are a prime example of this responsiveness, in their being organized around an end-to-end focus on networking in the large. The reorganization of AC&CSD to address the growing appearance of HPC-like functionality in everyday battlefield computing resources is another example. Further, within the portfolio of research projects there have been positive, significant changes, with projects dropped in areas that ARLTAB suggested were redundant or behind the state of the art (such as nanoelectronic devices), and new projects introduced in areas where there was evidence of significant Army mission-relevant potential (such as embedded HPC, networking problems, and bio-inspired applications). This responsiveness has even shown up in the way that individual divisions, especially BED and ISD, report out their research portfolios at the panel reviews. There is, however, still room for improvement, especially in articulating both divisional and overall CISD strategic plans and the rationale behind how the research portfolio is adapted to customer pressures while maintaining a solid and relevant basic science capability. There has been improvement, but it is not consistent across divisions, and the research portfolio does not roll up as crisply as it could to the strategic plans. An emphasis on defining the core long-term, relevant scientific problems and an articulating of short versus long-term strategic goals would help to maximize the value of CISD’s research portfolio to the Army. A suggested additional metric is how CISD’s customers perceive the value of their collaborations with CISD, with a related discussion of how expectations and requirements are developed in light of such metrics. As noted in the previous ARLTAB report, although there seems to be a significant number of collaborations of various sorts, it is often not clear how those collaborations interact with ARL programs (versus simply being funded grants), and what part of the results reported from the collaborations are due to ARL versus external researchers and contractors. This matter is important when trying to judge the overall level of expertise of the ARL staff. The final criterion asks whether a reasonable part of the ARL portfolio is being applied to breakthrough innovations as opposed to incremental progress. Although it is unclear what is reasonable, it is very clear that potential breakthrough innovations are being fostered in CISD. The Quantum Ghost Imaging work discussed above is one example presented during this review cycle.