7
Evaluation of the Planning Process

INTRODUCTION

Planning stages of a potential the Multifunction Phased Array Radar (MPAR) program for its entire life cycle from inception through decommissioning would include risk reduction studies, needs assessment, requirements definition, business case development, cost estimation, cost risk analysis, cost-benefit-analysis, appropriations, budgeting, research and development, full-scale acquisition, production, test, system integration, phased deployment, and operation. Initial planning for MPAR is in progress; the MPAR planning process is not static, but continuously evolves. Consequently, any evaluation of the process, however comprehensive, is at best a snapshot in time.

Chapter 6 and Appendix D of the Joint Action Group/Phased Array Radar Project (JAG/PARP) report summarize the proposed Research and Development plan developed by the Joint Action Group formed by the Office of the Federal Coordinator for Meteorology (OCFM). Emerging elements of the planning process to date, which will culminate in an MPAR Program Plan, are described earlier in the present report as presented to the committee.

Included within the MPAR Program Plan will be the MPAR Research and Development (R&D) Plan. This chapter evaluates the planning process to date and provides recommendations for its improvement. The committee recognizes that some planning elements that are identified as either deficient or missing may be implemented before the publication date of this report.

PURPOSE OF THE MPAR PLANNING PROCESS

The MPAR planning process, if successfully executed, will provide stakeholders and policy makers with reliable and sufficient evidence to support or reject a decision to proceed with the project at key decision points. Among these key decision points are whether to proceed with a risk reduction research and development program, whether to proceed with development of MPAR and T-MPAR prototypes, and whether to proceed with full-scale acquisition, development and phased production and deployment of MPAR.

The JAG/PARP report and additional materials and accounts of activities presented to the committee provide an opening round of MPAR planning that touches on some of the areas listed above. However, significant further work and strengthening of the planning process itself is needed.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 41
7 Evaluation of the Planning Process INTRODUCTION Planning stages of a potential the Multifunction Phased Array Radar (MPAR) program for its entire life cycle from inception through decommissioning would include risk reduction studies, needs assessment, requirements definition, business case development, cost estimation, cost risk analysis, cost-benefit-analysis, appropriations, budgeting, research and development, full-scale acquisition, production, test, system integration, phased deployment, and operation. Initial planning for MPAR is in progress; the MPAR planning process is not static, but continuously evolves. Consequently, any evaluation of the process, however comprehensive, is at best a snapshot in time. Chapter 6 and Appendix D of the Joint Action Group/Phased Array Radar Project (JAG/PARP) report summarize the proposed Research and Development plan developed by the Joint Action Group formed by the Office of the Federal Coordinator for Meteorology (OCFM). Emerging elements of the planning process to date, which will culminate in an MPAR Program Plan, are described earlier in the present report as presented to the committee. Included within the MPAR Program Plan will be the MPAR Research and Development (R&D) Plan. This chapter evaluates the planning process to date and provides recommendations for its improvement. The committee recognizes that some planning elements that are identified as either deficient or missing may be implemented before the publication date of this report. PURPOSE OF THE MPAR PLANNING PROCESS The MPAR planning process, if successfully executed, will provide stakeholders and policy makers with reliable and sufficient evidence to support or reject a decision to proceed with the project at key decision points. Among these key decision points are whether to proceed with a risk reduction research and development program, whether to proceed with development of MPAR and T-MPAR prototypes, and whether to proceed with full-scale acquisition, development and phased production and deployment of MPAR. The JAG/PARP report and additional materials and accounts of activities presented to the committee provide an opening round of MPAR planning that touches on some of the areas listed above. However, significant further work and strengthening of the planning process itself is needed. 41

OCR for page 41
42 EVALUATION OF THE MPAR PLANNING PROCESS Recommendation: The WG-MPAR Planning Process for the MPAR R&D program should implement frequent updating and improvement of the MPAR program plan to ensure planning robustness and relevance in the face of changing external conditions. As part of this process, the Program Plan should be periodically evaluated against program goals and objectives to ensure that they are both fully satisfied and remain relevant, as well as against the accomplishments of the R&D work. This evaluation should include annual external reviews, as suggested by Recommendation 3.5-6 of the JAG/PARP report. THE MPAR STAKEHOLDERS The OFCM is the primary executive for the MPAR program as now constituted. Prime stakeholders are the Federal Aviation Administration (FAA), National Oceanographic and Atmospheric Administration (NOAA) (including the National Severe Storms Laboratory [NSSL] and the National Weather Service [NWS]), the Department of Homeland Security (DHS), and the Department of Defense (DOD) (including the US Air Force, US Navy and US Army). Secondary MPAR stakeholders include the Federal Emergency Management Agency and the US Coast Guard, the Federal Highway Administration (FHWA), National Aeronautics and Space Administration (NASA), the Departments of Agriculture, Energy and Interior, and the Office of the Director of National Intelligence, as well as the NextGen Joint Planning and Development Office (NextGen JPDO). Primary stakeholders that have a stated commitment to fund MPAR research and development activity are: • FAA—for a cost-effective backup to the next-generation cooperative surveillance system and a possible replacement for legacy radars; FAA is currently funding phased array radar R&D. • NOAA—for continued funding of the National Weather Radar Testbed and additional funding for a MPAR risk reduction program beginning in FY 2010. As yet, neither DOD nor DHS have promised to fund MPAR activities, although discussions are underway. Both the OCFM and FAA agree that the large cost of R&D for MPAR and the need for interagency harmonization of requirements will require extensive interagency collaboration and the eventual creation of an MPAR Joint Inter-Agency Program Office. EXTERNAL PRESSURES ON EXISTING AND EMERGING MPAR STAKEHOLDERS As noted in the JAG/PARP report, planning for MPAR is driven by many factors including the rising Operations and Maintenance (O&M) costs of legacy radar platforms and societal expectations for improved performance in weather surveillance. It is also affected by the emergence of the NextGen and its reliance upon cooperative transponder

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 43 technology, known as ADS-B, as the primary means of civil aircraft surveillance. As the traditional role of air traffic surveillance radar correspondingly changes to that of a backup system for ADS-B,1 the existing FAA surveillance infrastructure will likely incur additional pressure to justify or reduce its rising O&M costs. As noted earlier (see also Box 7.1), this is already happening in the case of the Air Route Surveillance Radar (ARSR) system. En route aircraft surveillance by radar is expected to decrease in FAA priority while terminal area aircraft surveillance by radar is expected to remain the same. Support and funding for en route surveillance by MPAR by the FAA could weaken over the long term in a competitive budgetary environment with competing national priorities. On the other hand, if ADS-B fails to meet requirements for surveillance of cooperative aircraft, the potential posture of FAA vis-à-vis requirements for a future MPAR system may change. Weather surveillance requirements of NOAA, NWS and FAA for a future NEXRAD and Terminal Doppler Weather Radar (TDWR) upgrade or replacement are among the principal drivers behind their support of MPAR. Shortcomings of the NEXRAD system, as well as capabilities and opportunities afforded by new candidate architectures including phased arrays, are discussed in an earlier the National Research Council (NRC) committee report (NRC, 2002). The TDWR suffers high maintenance costs and for this reason is a candidate for upgrade or replacement. NEED FOR QUANTITATIVE REQUIREMENTS AND SPECIFICATIONS FOR MPAR A clearly stated set of requirements is needed to develop various candidate architectures for MPAR. At present, the only clearly defined requirements in existence are those for the four classes of existing weather and aircraft surveillance services; by default, these define the baseline capability required. This is shown in the “Current Capability” column of Tables 2.1 and 2.2 of the JAG/PARP report (reproduced as Tables 3.1 and 3.2 herein). One candidate MPAR architecture to meet this existing requirement is described in Appendix B of Weber et al. (2005), with an updated version in Weber et al. (2007). The multiple stakeholder needs survey response described above and summarized in the “Future Need” column of Tables 3.1 and 3.2 is the first step in defining a set of fully vetted requirements for MPAR. 1 The FAA’s Surveillance/Positioning Backup Strategies Alternatives Analysis Report “recommends that the FAA retain approximately one-half of the Secondary Radar Network as a backup strategy ADS-B”. It also recommends that “terminal area primary radar coverage will not be reduced from current levels”.

OCR for page 41
44 EVALUATION OF THE MPAR PLANNING PROCESS BOX 7.1 The ARSR-4 Legacy System for En Route Surveillance The current ARSR-4 unattended radar platform, or a derivative of it or similar system, might be adequate to fulfill future requirements for backup and non-cooperative en route surveillance of civilian aircraft. This is a well understood low-risk option for possible deployment and replacement of much older ARSR models, beyond the 40 sites2—mostly around the perimeter of the continental US—where it is presently employed. The network of ARSR-4s was delivered and installed between 1993 and 1999. The radar is estimated to cost $6.5 million per system. The total program cost is $800 million to date, half of which has been funded by DOD. The ARSR-4s are now operated jointly by the FAA and U.S. Air Force for air defense and drug interdiction operations, in addition to en route surveillance. The North American Air Defense (NORAD) fuses its data with that from other sensors for a common air defense picture of the continental United States (Forecast International Inc., 2003). The O&M responsibility for the ARSR network, as well as upgrades to it, have already been assumed by the DOD and DHS (Weber et al., 2007). As would be expected, the performance, reliability and maintainability of the ARSR-4 represent a significant advance over its predecessor systems. It employs a phased primary feed (a form of phased array technology) that provides a stack of 2-degree-elevation receive beams for simultaneous tracking of multiple targets in three dimensions. The JAG/PARP report (Table 3-1) estimates the ASR-4’s end of life in 2020, but this may assume that the government (DOD and DHS) will no longer fund FAA to upgrade the systems or replace service parts (Forecast International Inc., 2003). On the weather surveillance side, the ARSR-4 has been identified as having future FIGURE 7.1. Photo of an development potential as a “gap-filler” for the ARSR-4. Source: Northrop- NEXRAD network (Istok, 2005). Grumman Corporation. 2 A total of 61 earlier generation ARSR-1/2/3 radars are sited around the country and will eventually need to be replaced.

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 45 Where these needs exceed present capabilities, a careful exercise of architectural trades should be performed including Cost as an Independent Variable (CAIV).3 From this exercise, a realistic set of requirements could be generated. From these requirements, various candidate architectures could be proposed and one or more could be selected for further evaluation. For the winning architectures, conceptual designs could be developed and further refinement leading to the selection of an optimum design could be accomplished. A prototype would then be built to the optimum design for further testing and refinement. Two stated “Future Needs” (listed in Table 3.2), for the sensitivity of non- cooperative airborne surveillance at 0.1m2 radar cross section and vertical coverage from the surface to 100,000 ft, are examples of where a possible CAIV design trade could balance cost against achievable performance. Candidate architecture to satisfy these two needs for homeland defense would be particularly costly if 100 percent coverage of the Nation’s borders is to be achieved. The architecture proposed in Appendix B of the JAG/PARP report would not satisfy either of these two stated needs, even in areas where full scale MPAR coverage is provided. Other Potential Requirements The task of defining the evolving spectrum of “air-breathing” threats for homeland defense is the responsibility of the Joint Air and Missile Defense Organization (Evans, 2004; Mathis, 2004). This is the organization within the DOD chartered to plan, coordinate, and oversee Joint Air and Missile Defense (AMD) requirements, joint operational concepts, and operational architectures. An expanding but as yet only preliminarily defined role for future radars to provide surveillance against a variety of non-cooperative airborne threats for Homeland Defense could become a significant factor in determining MPAR requirements. The current baseline air surveillance capability against such threats is inadequate, particularly at low altitudes. The present MPAR plan for some 334 systems appears only to address the current FAA baseline surveillance requirement for commercial aviation. If, as seems likely, significantly more than 334 MPARs should be deemed necessary to provide comprehensive low altitude coverage for homeland defense, then even if the aggregate cost projections were accurate, they would only apply to an established FAA requirement rather than a new DHS/DOD requirement. An alternative approach similar in concept to the dense, low cost, low power CASA radar architecture (see Chapter 8) could possibly be investigated for seamless low altitude coverage of airborne threats crossing the nation’s borders. Section 6.4.3 of the JAG/PARP report indicates that “coordination and collaboration with the CASA program will be essential to this part of the risk reduction program.” The relationships between the proposed MPAR system and potential future 3 For a discussion of CAIV and its relationship to Total Ownership Cost (TOC) or Life Cycle cost, See Boudreau, M. W. 2005. “Total Ownership Cost Considerations in Key Performance Parameters and Beyond.” Defense Acquisition Review Journal, Feb – March. Available at http://www.dau.mil/pubs/arq/2005arq/2005arq-38/boudreau.pdf accessed July 24, 2008.

OCR for page 41
46 EVALUATION OF THE MPAR PLANNING PROCESS needs, and a potential future system based on the CASA concept, are unclear. Indeed, the relationship between the MPAR and CASA projects is unclear and needs to be clarified to understand how requirements, benefits, and costs are being assessed. Looming upon the horizon are two national directives that have the potential to significantly influence the technical direction and scale of the MPAR program: National Security Presidential Directive 47 and Homeland Security Presidential Directive 16. These two Presidential directives direct the production of the National Strategy of Aviation Security and several supporting plans including the Air Domain Surveillance and Intelligence Integration Plan4 that address surveillance radars (which would include MPAR) as a collection source for National Intelligence to promote the goal of Air Domain Awareness. Recommendation: The MPAR R&D program should produce a fully vetted set of technical performance requirements for an operational MPAR and radar network. To ensure robustness of the R&D Program in the face of potential re-balancing of stakeholder needs and participation over time, the MPAR planning process for non- weather surveillance should further emphasize the need to fully establish requirements of all participating agencies. EVALUATION OF THE MPAR R&D PLANNING PROCESS A Research and Development plan is an essential component of the MPAR planning process. Such a plan is needed to reduce identified key technical risks or potential “show stoppers” (such as whether a phased array radar can effectively perform dual polarization measurements) and other issues identified in Chapter 5 above and Chapter 4 of the JAG/PARP report that, if not mitigated, would effectively halt a decision to proceed with further MPAR development. A second goal of the R&D plan as identified in the JAG/PARP report is the “establishment of a documented basis for cost comparisons between the MPAR and mechanically rotating conventional radar (MRCR) alternatives for meeting national domestic radar surveillance needs…” This goal is not fully supported by the stated R&D activities, but the committee views it as essential to providing the basic business case for whether or not to proceed with an MPAR acquisition. TECHNICAL ISSUES The proposed R&D program addresses many of the technical risks listed in Chapter 5 above, but not other essential capabilities, such as “demonstrating the operational capability enhancements that can be realized through collaborative 4 See http://www.dhs.gov/xlibrary/assets/hspd16_domsurvintelplan.pdf.

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 47 surveillance strategies that exploit the unique capabilities of a highly interconnected phased array network.” Table 6.1 in the JAG/PARP report (reproduced below; see Table 7.1) summarizes the key MPAR “technical parameters” posing the most significant challenges that should be addressed by the R&D program. Twelve critical tasks are listed in the Technology Development and Test Program. Successful execution of the first five tasks constitutes the first go/no-go decision point. TABLE 7.1. MPAR Key Technical Parameters. Source: OFCM, 2006. Total Number T/R-Elements per Radar Number of Frequency Channels Dual Polarization Bandwidth (per channel) T/R-Element Peak Power Number of Concurrent Receive Beams Software Complexity Size, Weight Constraints Prime Power Constraints NOTE: The background colors denote the level of technical and/or cost challenge imposed by each parameter. Red denotes substantial challenge, yellow denotes moderate challenge, and green denotes minimal challenge. The capability of phased array radars for aircraft surveillance and tracking has been well established through several decades of military experience. However, the same cannot be said for the capability for weather surveillance—especially for the quantitative measurements required for most effective use of the observations. Suitability of an MPAR system for weather surveillance will be predicated on achieving measurement capabilities comparable to those that will exist in the NEXRAD and TDWR systems at the time any decision to proceed with implementation of an MPAR network must be made. This includes not only the present capability for reflectivity and Doppler velocity measurement, but also the soon-to-be-deployed NEXRAD polarimetric capability. Moreover, the narrower beamwidth and higher frequency of TDWR provide capabilities differing from those of NEXRAD. The narrower beamwidth of the TDWR provides sufficient vertical resolution to observe shallow air motions above airport runways, reduces ground clutter, and provides high resolution weather measurement in the terminal area. A requirement to retain these capabilities would have substantial impact on an MPAR or T-MPAR design.

OCR for page 41
48 EVALUATION OF THE MPAR PLANNING PROCESS Reflectivity Quantitative reflectivity measurements to accuracy of 1 dB (the NEXRAD Technical Requirement goal) require careful system calibration, a topic notably absent from the table. In an MPAR system the characteristics of the antenna beam, including the beamwidth and antenna gain, vary with squint angle and may also differ between transmit and receive modes. Although in principle these variations can be determined and accounted for, a means of verifying these corrections in field-deployed systems will be needed. Measurement of transmitted power and receiver response characteristics is also challenging with a distributed array. The R&D planning for MPAR must include development of a practical approach for dealing with this problem in an operational environment. Preliminary testing of the approach could be carried out at the NWRT, but the committee received no information about any such effort. Polarimetric Variables An MPAR system could offer some advantages over the NEXRAD polarimetric capability—for example, the possibility of obtaining LDR measurements. However, the polarization characteristics of the antenna beam vary with squint angle. Part of the variation results from simple geometric considerations, and here again the variations can in principle be determined and accounted for. However, other factors such as mutual coupling between array elements at large squint angles complicate the situation and are not so readily analyzed. This represents a major challenge to the suitability of MPAR for weather surveillance, and the challenge in verifying any specified procedure with field- deployed systems is even greater than the reflectivity problem. The MPAR R&D planning process should include a well-developed concept for evaluating the polarization capabilities of real MPAR systems. Recommendation: The MPAR R&D program should produce a procedure for calibrating the reflectivity and polarimetric measurements at all scan angles. A key decision point for the feasibility of MPAR for weather surveillance, and continuance for the R&D program, will be determination of its capability for dual polarization measurements. Therefore, thorough evaluation of the capability of phased array radar to accurately measure polarization variables independent of scan angle must be carried out early in the R&D program. Frequency Requirements The MPAR R&D program should address the frequency allocation/interference issue. Given the current design concept proposed to accomplish the multi-function capabilities (Weber et al., 2007), independent transmitter frequencies will be needed for each function in order to provide the necessary time-on-target. In addition, use of pulse compression methods may require additional “fill pulses” to provide short-range

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 49 coverage, necessitating yet another frequency allocation (as well as increasing the needed dwell times). The MPAR would have four faces, essentially four separate radars, and it is possible that each face would require an independent set of frequencies in order to eliminate face-to-face interference. In a worst-case scenario, the full four-face MPAR could require as many as 16 frequencies, with associated bandwidths and guard-bands. Access to the required frequency spectrum in S-band, where the commercial demand for spectrum allocations is ever increasing, could become a challenge to MPAR implementation. This factor must be considered in the planning process. Recommendation: Given the high demand for bandwidth at the proposed S-band frequency, the MPAR R&D program must determine the total required bandwidth as early as possible in the research program to ensure the feasibility of the design. T-MPAR Planning Process The conceptual system design described by Weber et al. (2007) suggests that a national MPAR network may consist of two radar types. Approximately one-half would be full-scale MPARs with maximum sensitivity, resolution, and operating range; the other half would be less-costly “Terminal-MPARs” (T-MPARs) with smaller apertures. The full-scale MPARs would be used as the Next Generation Radar (NEXRAD) replacements and at airports currently served by the TDWRs. The T-MPARs would be used, as a lower cost option, at smaller airports and as gap fillers. As noted in the JAG/PARP report and in Weber et al. (2007), this is a preliminary and “not fully worked out” design concept. The adequacy of this concept, ranging from the ability to track aircraft to the required weather sensing capabilities, should be fully scoped out. The T-MPAR transmit/receive (T/R) modules could be designed to operate at S- band (as for the full MPAR), or possibly at C- or X-band. A T-MPAR design operating at S-band5 would achieve commonality of T/R modules across the full national system and maximize the economies of scale achieved through mass production. If shorter- wavelength options appear desirable, economies of scale may be compromised in both the production phase and later in the operations and maintenance phase after deployment. The FAA’s surveillance roadmap identifies a “New Primary Radar” to replace the ASR-8/9/11 terminal platforms around the year 2020. The T-MPAR could be a candidate for this new “Primary Radar.” As noted in Weber et al. (2007): “T-MPAR would be deployed primarily at smaller airports where today, either wind shear protection services are not provided, or are provided by the less capable ASR-9 Weather Systems Processor.” This suggests a lower level of terminal weather surveillance performance for certain regions of the country. For example, if modules having the same transmit power are used for both MPAR and T-MPAR, the total transmitted power for the T-MPAR would be reduced by the ratio of the antenna aperture areas. Consequently, the power-aperture product of T-MPAR would be reduced (compared to MPAR) by the square of the ratio of the aperture areas. 5 Statement by Mark Weber of Massachusetts Institute of Technology/Lincoln Laboratory at the committee’s second meeting in Boulder, Colorado on March 6, 2008.

OCR for page 41
50 EVALUATION OF THE MPAR PLANNING PROCESS Section 2.3 of Appendix B of the JAG/PARP report notes, for one design of a dual- density active array for the “gap filler” radar, that “Although possibly adequate for precipitation mapping and many Doppler measurement applications, this gap-filler configuration would be substantially less sensitive at short range than are current TDWR or NEXRAD systems.” Therefore, the suitability of the T-MPAR design for meeting functional requirements must be fully evaluated. Without a clearly defined set of requirements for MPAR, or a completed design concept for T-MPAR, one cannot state whether one or two radar types is the preferred (or indeed, lowest-cost or optimal) approach. Recommendation: The Airport Terminal Area or T-MPAR concept needs to be developed in sufficient detail to demonstrate that mission requirements for terminal weather and aircraft surveillance can be met. In addition, the ability of a full MPAR to meet Terminal Doppler Weather Radar (TDWR) requirements must also be assessed due to the fact that the MPAR beamwidth would be approximately 1 deg (instead of ½ deg) and the frequency choice is S-band (instead of C-band). COST ISSUES A number of assumptions underlie the preliminary MPAR cost estimate provided in the JAG/PARP report (Chapter 5 and Appendix C). The pre-prototype component cost estimates presented in Table 2 of Appendix C are extrapolated to full-scale MPAR component costs based on “economies of scale or new technologies expected to mature over the next three years.” These advances are no doubt likely to happen to some degree (as noted in the section below on element cost), and the effects may be describable via parametric relationships such as the “learning effect.” However, it is difficult to quantify the size or sensitivity of the parameters or to predict the level of cost reduction that will be achieved by these and other advances in the future (as is implied in the cost column titled “Full Scale MPAR”). A key question seems to be: “what is the probability of deploying MPAR at or below the projected cost of $3.34 billion?” Only after a prototype is developed and evaluated does it seem possible to be more specific about the overall cost of MPAR. Approximately one half of the MPAR radars are projected to be the smaller and lower-cost T-MPARs providing limited coverage underneath the radar horizon of the national-scale network. The mix of short-range and long-range MPARs in the final network configuration appears to be a major driver of costs and therefore cost estimate uncertainties. A more thorough and systematic approach to cost estimation is needed; various DOD-related publications present an introduction to modern cost estimation methodology (Book, 2001). Cost of Array Elements Table 2 in Appendix C of the JAG/PARP report shows a target price for T/R modules of $20 each in a production MPAR system. The cost of T/R modules has long

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 51 been seen as prohibitive to the use of phased array radars in applications other than high- performance military missions. A compelling argument presented to the committee by commercial representatives is that through careful electronic design and judicious choices of electronic materials, foundry, physical packaging and production line, a T/R module cost in quantity of below $50 is achievable. This argument is based on cumulative production experience in developing Radio Frequency (RF) componentry on commercial lines for mass-produced portable wireless handsets, radio frequency identification readers and automotive radars. It is argued that commercial packaging and production techniques used for these high-volume commercial products would permit similar economies of scale to be achieved for MPAR. The most likely cost that will be achieved for MPAR T/R elements is difficult to predict with precision, but this one item will receive considerable attention from the system designers and cost modelers. A key consideration is the question of whether a common T/R module design will suffice for both the MPAR and T-MPAR systems. Another major consideration is the cost and added complexity associated with T/R module design involving dual polarization and multiple frequencies. From the standpoint of total system cost, the difference between $20 and $50 for a T/R module is highly significant. An extra $30 per T/R module equates to an extra $2.4 million per MPAR. Further uncertainties in other hardware costs will only add to the possible range of total cost per system. The MPAR cost analysis as presented in the JAG/PARP report only addresses an MPAR architecture that meets the baseline requirements. Where an entirely new and unproven architecture is presented that meets but does not substantially exceed the performance baseline, a compelling and robust cost-reduction argument would need to be presented; this has not yet occurred. The basic cost-saving argument centers upon providing coverage, from a reduced number of fielded MPARs and T-MPARs, that is essentially the same or marginally better than that presently provided by the existing network of legacy radars. From Table 2 of Appendix C, the target total cost of all the electronics normalized per T/R element (including $20 for the T/R module itself) is $133.50. From this basic building block, the total cost for a full 80,000 element MPAR is projected to be $10.7 million and the T-MPAR with about 8,000 elements to cost $2.8 million. These cost estimates are at best rough order of magnitude estimates and, as they stand, are inadequate to form the basis of an informed procurement decision. These could be optimistic figures and will need to be revisited throughout the R&D risk reduction program and beyond to ensure that a viable economic argument for fielding MPAR can be made. With the prospect of a production run of hundreds of nearly identical radar systems and millions of T/R modules, it would be useful to engage multiple sources to manufacture and deploy MPAR and T-MPAR systems, sub-systems and components. Competition can be used advantageously both to minimize cost and to enforce the development of and strict adherence to an open-standards-based architectural framework. As noted earlier, an open-standards-based approach is preferable to an architecture that is based on closed, proprietary standards from a single contractor. Future expansion and enhancement of MPAR would be greatly facilitated by open standards. The JAG/PARP report correctly notes the value of open architecture in the use of commercial off-the-

OCR for page 41
52 EVALUATION OF THE MPAR PLANNING PROCESS shelf software (section 4.5) and hardware (section 5.1). However, this needs to be an enforced requirement that is written into the acquisition contract. Federal authorities could require redundant MPARs to be installed in overlapping coverage areas or locations deemed of critical importance. The consequences of losing both weather surveillance and backup air-traffic surveillance from the failure of a single radar could be deemed risky. Fielding additional MPARs to reduce this “all eggs in one basket” risk could significantly increase the number of fielded systems above the 334 proposed and therefore reduce the economic advantage cited. The O&M costs for the entire population of legacy radars are estimated, based upon Lincoln Laboratory’s involvement with the life-cycle support and enhancement programs for TDWR, NEXRAD and ASR-9 programs, at $0.5 Million per unit per year. It was stated to the committee that the TDWR and NEXRAD radars have the highest maintenance costs6 because of high rates of wear on the azimuth and elevation antenna drive axes. If these three radars form the basis of an O&M cost estimate for legacy radars that is then extrapolated to include the newer ASR-11 and ARSR-4 systems, which require less maintenance, these costs could be incorrectly averaged over the wider legacy radar population. It is possible that the same O&M savings could be realized with less investment and lower risk by replacing just the two weather surveillance radars with a new design, and simply replacing the least reliable legacy aircraft surveillance radars with newer models. The O&M cost estimates for MPAR are assumed to reduce to $0.3 Million per radar per year, based in part on the synergy effects caused by a reduction in the required number of program offices, personnel, and non-recurring engineering (NRE) expenditures. This optimistically assumes a steady-state condition after transition costs associated with a complex changeover to a new system. However, a period of overlap would likely occur between the deployment and operation of each new MPAR and the decommissioning of the legacy radar at each operational site. Integration of the new radar into a new or modified legacy network would likely consume additional time and effort at additional cost, particularly if unforeseen problems arise in the later stages. The current assessment of the potential cost paths with MPAR versus MRCR, as illustrated in Figures 3 and 4 of Appendix C of the JAG/PARP report, does not account for the fact that the likely start date of any MPAR system implementation would lie a decade or more into the future. This assessment does not indicate a net present value based on common methods for discounting and accounting for risk. A more complete and acceptable net present value should be presented, in conjunction with a sensitivity analysis for key unknowns—such as uncertainty with respect to the likely cost of T/R components. Recommendation: A thorough and complete cost analysis of the total MPAR program should be performed and compared with historical life-cycle costs for the more recently and currently deployed systems such as ARSR-4 and ASR-11 that are roughly equal in performance to MPAR for air-traffic surveillance, and for NEXRAD and TDWR radars that provide a performance baseline vs. MPAR for weather surveillance. A detailed baseline operations and maintenance (O&M) cost estimate should be 6 By William Benner, FAA, at a presentation to the committee on January 14, 2008.

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 53 determined for all legacy radar types to identify and quantify those highest cost radar types that are the prime candidates for life extension, upgrade or replacement. Independent cost risk analyses for the acquisition of MPAR and T-MPAR by recognized methods should be performed and continuously re-visited and updated. Recommendation: An alternative weather-only phased array weather radar design trade study and detailed cost analysis should be performed and compared with historical life cycle costs and performance for NEXRAD and TDWR radars. This trade study and cost analysis should be compared with a more detailed MPAR cost analysis and trade study to determine if the marginal cost of adding the required aircraft surveillance capability is worth the perceived benefit of having an all-in-one system. NEED FOR COMPLETE INCLUSION OF ALL ASSOCIATED SYSTEM COSTS The cost estimates for MPAR in Appendix C of the JAG/PARP report are based solely on component hardware costs, normalized to cost per T/R module. Software development costs (internal, application and system), integration cost, site preparation, deployment, testing, management and other NRE design costs are not considered. For a large production run of a mature product, assuming that design changes are not implemented during the production run, early-stage NRE would be averaged into the cost of individual components. Thus, total component and production costs would eventually dominate the cost per radar. However, at the early stages of the project, research and development expenses (including software development activity) would dominate the cost picture. Other cost issues raised in Chapter 5 would affect the life-cycle costs of MPAR. Accounting for these issues should include evaluation of not only technical issues, but also logistical (e.g., frequency allocation and siting) and implementation (e.g., education and training) issues. For example, if parallel operation of the legacy system with the new MPAR system is required for some limited time to transition smoothly from one to the other, then the site occupied by the legacy system cannot be used by the MPAR system. Also, it would be overly optimistic to assume that extensive software development activity will not extend well into the operational life of an MPAR program. The experience with NEXRAD suggests that it will more likely extend for years beyond the initial deployment period. MPAR cost estimates should include all likely software development, integration, testing and upgrade costs beyond initial operational capability that extend from the initial first fielded system through final deployments up to a defined baseline level of individual radar and fully integrated system-wide performance COST-BENEFIT ANALYSIS Recommendation 4 of the JAG/PARP report states: “The FCMSSR should direct that, in conjunction with the MPAR risk-reduction program, a cost-benefit analysis be undertaken to establish the cost-effectiveness of the MPAR option and competing domestic radar strategies. The basis for MPAR acquisition and life-cycle costs should

OCR for page 41
54 EVALUATION OF THE MPAR PLANNING PROCESS include results from the technology development and test activities and the MPAR network refinement, as appropriate.” We note that the purpose of the cost-benefit analysis is not to establish the cost-effectiveness of a program but rather to quantify the net benefits of that program and potentially compare those to the net benefits of a baseline or alternative programs. Chapter 2 of the JAG/PARP report discusses a wide range of potential needs that radar could meet, including weather surveillance, aircraft surveillance, and a variety of other uses. None of these uses—i.e., benefit areas—are quantified or monetized as part of a benefits assessment. Support for the MPAR R&D project in the JAG/PARP report is based entirely on a comparison of projected costs between MPAR and MRCR, rather than on anticipated benefits compared to costs of the various alternatives. As there is little available information on the economic benefits of the current radar systems, the R&D project should include a research component to identify the current and potential communication, perception, use, and values for radar-based information for a broad range of users. This would include both weather and aircraft surveillance functions and a broad spectrum of users including different economic sectors and subsectors (such as transportation, energy, agriculture, or insurance), public sector users (such as emergency management, water resources, environmental management, aviation, or homeland security), and the public at large. Economists have developed methods to address the challenging task of monetizing seemingly intangible benefits, such as public safety and saving lives. For example, the value of a statistical life (VSL) is used to estimate the monetary benefit of reducing premature mortality risks using available willingness to pay (WTP) estimates for changes in mortality risks on a per-life- saved basis. VSL is a theoretically valid and widely used measure for evaluating the benefits of programs that affect mortality rates, including environmental protection issues (Dockins et al., 2004). As the project develops, an ongoing effort is needed to track, assess, quantify, and monetize the benefits and costs as these develop over the course of the effort. This will provide ongoing feedback assessing the net benefits of the project and identifying areas of highest potential benefits to focus potential research and applications efforts. Ongoing benefit-cost assessment will also account for unanticipated changes in technology, societal needs, and newly developed or identified application areas that could generate previously unaccounted benefits. The design of the MPAR system as described to the committee appears to be based entirely on functions provided by the current system. Unless there are binding constraints requiring maintaining the current coverage, an assessment of the future of the U.S. radar system should be based on an optimal design for future needs. Whether this means more or less coverage (or the same) compared to the current system should be determined in an empirical assessment of system needs. This requires a thorough assessment of the benefits and costs of the current system and likely future requirements based on spatially optimal needs in a benefit-cost analysis framework.

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 55 Cost of the R&D Program The $215 million R&D plan, as originally envisioned and described in Appendix D of the JAG/PARP report, is divided into three major areas that were to be executed in parallel beginning in FY07 and ending in FY15: • A total of $52 million is to be spent on “Proof of MPAR Operational Concepts” at the NWRT in Norman OK. • A total of $5 million is to be spent on “Refinement of MPAR Network Concept” for X-band and C-band dual-polarization phased array “Gap Filler” radar development that “would compliment the CASA research”7 (MPAR, and probably T-MPAR, would operate at S-Band). • A total of $158 million is to be spent on “MPAR Technology Development and Test,” including the MPAR architecture study; development of T/R modules and subsystems; and an MPAR pre-prototype and full-scale prototype. TABLE 7.2. JAG/PARP Report Budget FY07-FY15 7 R&D funding clarification memo. March 19, 2008, Jeff Kimpel, Director, National Severe Storms Laboratory, Norman, OK.

OCR for page 41
56 EVALUATION OF THE MPAR PLANNING PROCESS NOAA provided supplementary budget details for the MPAR R&D program, reproduced in Table 7.2 below. We note that the FY07 and FY08 funding provided has been insufficient to fully execute the R&D plan as outlined in the report. Recommendation: The FCMSSR should seek a reasonable and continuous funding stream to support the R&D Program. Finding The planned expenditures in the second major area are aimed at developing new dual-polarized mechanically- and electronically-steered radars at C and X band, and analyzing data obtained with those radars to support possible alternative configurations to multifunction S-band arrays. In contrast to the S-band MPAR work, the specified funding levels and range of activities represent only a fraction of the research needed for risk reduction at shorter radar wavelengths. Risk reduction activities in support of X- band phased array radar, signal processing, and data communication technologies are currently being carried out by the CASA Engineering Research Center with support from the National Science Foundation. The committee appreciates the intent of the MPAR activity to link with the CASA center but notes that no specifics are given on how that linkage would be made. The committee has some concern that many of the short- wavelength activities described appear to duplicate some of the efforts of CASA and other projects. The latter include a number of well-calibrated dual-polarized mechanically-steered radars that currently exist within the remote sensing community and that can support new phenomenological investigations. Better utilization of these capabilities can be a more effective approach to achieving some of the MPAR R&D goals than developing an entirely new set of C-and X-band radar systems. Recommendation: The MPAR R&D Program, instead of developing new X- and C- band radars, should develop linkages with appropriate organizations within the radar community as a way to avoid duplication of effort and take full advantage of ongoing work related to short-wavelength radar technologies. The Full Scale Prototype Stage Of the $158 million to be spent in the third major area, the cost of the MPAR pre- prototype stage is $18 million, whereas the full-sized MPAR prototype and subsequent testing stage totals $140 million. Chapter 6 of the JAG/PARP report states that tasks 1 through 5 of the list in Section 6.2 pertain to a pre-prototype MPAR, where the most critical technical issues are to be addressed earliest at the lowest cost. A decision to proceed with full-scale prototype development will be required before the bulk of planned MPAR R&D funding needs to be spent. The full-scale prototype is only needed for tasks 9 through 12, and only after major decision hurdles have been met. The committee senses that the majority of R&D issues associated with MPAR surveillance of aircraft and weather could be addressed with a prototype single full-scale

OCR for page 41
EVALUATION OF THE PLANNING PROCESS 57 antenna face, or possibly two faces (to examine handoff issues and continuity of calibrations across faces). A capability to position this face to all azimuths would be useful. A capability to transport the prototype platform could also be useful to allow investigating MPAR capabilities in a variety of weather and air traffic regimes. Recommendation: The MPAR R&D program should include the staged development of a prototype MPAR, proceeding through a Line Replaceable Unit (LRU), followed by a single antenna face, two faces, or a full four-faced prototype. Cost effectiveness studies should be carried out to determine how many faces would be required to assess the MPAR concept. Recommendation: The committee endorses Recommendation 2 of the JAG/PARP report and would like to see it implemented early in the program. The committee further recommends that the MPAR R&D program be as open as possible, in particular to ensure that interested parties from industry and universities are involved at early stages, and that the engineering development and scientific applications of the MPAR prototype benefit from involvement of the broadest communities possible. Cost-Benefit Analysis of the R&D Program No direct link is made in the JAG/PARP report between the discussion of benefits and costs of a fully implemented MPAR system and the R&D program, in terms of the assessment of the full system providing information on the potential benefits of the R&D program. Realizing the potential benefits of the full MPAR system is dependent on success in the R&D program, and one purpose for assessing the potential costs and benefits of an MPAR system is to provide information for a cost-benefit analysis of the R&D program itself. A valid cost-benefit assessment of the R&D program requires sufficiently detailed and supported information on the likely cost of the R&D effort and probabilistic assessment of cost uncertainties. Appendix D of the JAG/PARP report outlines the MPAR R&D Plan with costs indicated in parentheses by each task-year. The total of $215 million is based on a series of estimates for sub-components of this project, but there is not adequate documentation of the source of these estimates. In response to an information request from the committee (“How were the estimates arrived at in Chapter 6 and Appendix D of the JAG report?”), James Kimpel provided the following response: “c. Based on the team members . . . experience (20 years plus for most members), an educated estimate of what it would take to accomplish each of the tasks was prepared. Some of the tasks use in-house expertise and some require contracting out. Some of the tasks required the purchase of state-of-the-art hardware, building a dual-polarized sub-array or a full sized Multi-function Phased Array Radar. d. Based on staffing and hardware requirements, the estimates were drafted and then refined several times. Expertise from NOAA, FAA, and Lincoln Labs all participated in refining the estimates.”

OCR for page 41
58 EVALUATION OF THE MPAR PLANNING PROCESS The information developed to date on costs appears inadequate as justification for the MPAR R&D effort. Given the magnitude of the proposed effort, more complete information is needed on the estimation of the costs of the MPAR risk-reduction program. Recommendation: The R&D Plan outlined in Appendix D in the JAG/PARP Report should be expanded to provide detailed descriptions of the tasks to be undertaken, their priorities, the associated costs, and key decision points. The major potential benefits from the MPAR R&D effort are the likely benefits from an MPAR system in whatever form that system is deployed. A cost-benefit analysis of the deployed MPAR system is needed, as described earlier, in order to determine the potential benefits from the R&D effort. These potential benefits would be weighted by the (subjective) probability that the R&D program would establish the viability of the MPAR alternative to MRCR. There is no discussion in the JAG/PARP report of probability estimates for success or failure at any critical decision points in the R&D program. Recommendation: Probability estimates of the likelihood of success/failure of achieving objectives at critical decision points in the R&D program should be developed. The discussion of cost savings of an MPAR program in the JAG/PARP report focuses on a future system implemented to replace the legacy systems. In addition to the need for a cost-benefit analysis (CBA) of the full implementation of an MPAR system based on results of the risk reduction R&D program, there needs to be a complete CBA of the risk reduction program itself prior to funding of the program. This CBA would assess the expected net benefits of the MPAR R&D program in relation to the proposed $215 million R&D cost. In order to accomplish this it is necessary to have baseline information on the expected long-term benefits from a future MPAR system. A CBA of the R&D program should consider a range of alternatives, including such things as partial replacement of the legacy radar system with T-MPAR and the potential benefits of investing in R&D to improve MRCR systems. The probability estimates of success could be derived from an expert assessment (Delphi method or other methods). These probabilities should then be updated if and as the R&D program proceeds and more information becomes available.