6
An Administrative Group for Project Review

An Administrative Group for Project Review (AGPR) should be established to organize and administer review of Corps of Engineers water resources planning studies and projects. The Administrative Group for Project Review would have three options for deciding if a Corps water resources planning study should be reviewed: (1) current review procedures, (2) internal review, and (3) external, independent review conducted by an organization independent of the Corps. In addition, a Review Advisory Board (RAB) should be established to provide continuing oversight and fresh perspectives for the Administrative Group for Project Review. This Board should be established and maintained by an organization independent of the Corps.

The Administrative Group for Project Review should consist of a relatively small, full-time, and permanent professional staff reporting directly either to the Assistant Secretary of the Army for Civil Works or to the Chief of Engineers. To ensure the highest quality of reviews, the director of the Administrative Group for Project Review should be a Senior Executive Service-level professional, and the professional staff should possess broad knowledge and expertise in disciplines relevant to water resources planning. The Administrative Group for Project Review would assess Corps reconnaissance reports in order to designate which future draft feasibility reports should be reviewed. The Administrative Group for Project Review would apply magnitude-risk criteria, such as described in Chapter 4, to select the appropriate type of review.

Another important AGPR responsibility would be to prepare a document summarizing the key assumptions, methods, and conclusions of a planning or reoperations study. This summary document would be



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 57
Review Procedures for Water Resources Project Planning 6 An Administrative Group for Project Review An Administrative Group for Project Review (AGPR) should be established to organize and administer review of Corps of Engineers water resources planning studies and projects. The Administrative Group for Project Review would have three options for deciding if a Corps water resources planning study should be reviewed: (1) current review procedures, (2) internal review, and (3) external, independent review conducted by an organization independent of the Corps. In addition, a Review Advisory Board (RAB) should be established to provide continuing oversight and fresh perspectives for the Administrative Group for Project Review. This Board should be established and maintained by an organization independent of the Corps. The Administrative Group for Project Review should consist of a relatively small, full-time, and permanent professional staff reporting directly either to the Assistant Secretary of the Army for Civil Works or to the Chief of Engineers. To ensure the highest quality of reviews, the director of the Administrative Group for Project Review should be a Senior Executive Service-level professional, and the professional staff should possess broad knowledge and expertise in disciplines relevant to water resources planning. The Administrative Group for Project Review would assess Corps reconnaissance reports in order to designate which future draft feasibility reports should be reviewed. The Administrative Group for Project Review would apply magnitude-risk criteria, such as described in Chapter 4, to select the appropriate type of review. Another important AGPR responsibility would be to prepare a document summarizing the key assumptions, methods, and conclusions of a planning or reoperations study. This summary document would be

OCR for page 57
Review Procedures for Water Resources Project Planning crucial to expediting the duties of review panels, as the details of Corps of Engineers planning studies may be difficult for reviewers—especially external expert reviewers—to quickly and thoroughly comprehend. The Administrative Group for Project Review would maintain all documents associated with a review, report periodically to the independent Review Advisory Board (discussed in more detail below), and act as the advocate for review. The AGPR should organize internal review panel meetings and activities and should assemble, publish, and disseminate reports from internal review panels. Reviewers should be selected on a project-specific basis, and the number of reviewers and the fields of expertise should be determined by the nature of the planning study. The Administrative Group for Project Review should develop and maintain a resource list of possible reviewers. External review reports should be published and disseminated by the organization conducting the study. The Administrative Group for Project Review should prepare a set of clearly written procedures that define the Corps’ review procedures. As the Corps gains experience with these procedures, the procedures will likely evolve. The written guidelines should be viewed as flexible and the AGPR should periodically revise these written procedures as necessary. Some reviews are likely to be complex and may require substantial commitments of reviewers’ time. In some circumstances, it may thus be appropriate for reviewers to be compensated. Suitable staff support for the review is also important. A large staff does not appear to be necessary, but the AGPR should include both a professional staff with a strong working knowledge of the Corps and its operations, including a director with Senior Executive Service status, and technically qualified senior staff. Congress should assure that the AGPR has the professional staff to execute all these duties. INSTITUTIONAL HOME The NRC Panel on Peer Review reached several conclusions that draw upon the options and criteria discussed in Chapter 5. There are options that give authority for review to professional engineering or science societies or to other groups beyond the Corps. In considering an institutional home for the Administrative Group for Project Review, however, these bodies are likely to be too administratively distant from the Corps and its planning studies to provide effective and sustained oversight of all review procedures. Those independent organizations are also unlikely to place adequate priority on the review process to allocate resources sufficient to sustain the activity. The Office of Management

OCR for page 57
Review Procedures for Water Resources Project Planning and Budget has its own unique role in the budgetary process, and therefore it is not an appropriate place to establish any type of Corps of Engineers administrative review group. This panel identified two viable options for the AGPR’s institutional home: the Office of the Assistant Secretary of the Army for Civil Works (ASA(CW)) and the Office of the Chief of Engineers. There is a long and rich history of the respective roles that each of these bodies has played in the review of Corps planning studies, as well as their relationship with one another. In discussing this history, our panel noted that these roles have shifted over time and that the balance of responsibilities for review between the two offices has also occasionally shifted. An example of such shifts is illustrated in a memorandum dated August 23, 2001, from Mr. Dominic Izzo, Principal Deputy Assistant Secretary of the Army for Civil Works. The memo describes the creation of a new group within the ASA(CW) Office to improve oversight of the project planning and review process (Appendix C). Future shifts in responsibilities for the review process are a distinct possibility, and these shifts will have implications for the appropriate institutional home for the Administrative Group for Project Review. These shifts also make discussions about an appropriate institutional home for the AGPR problematic. The panel concluded that neither the Office of the Chief of Engineers nor the Office of the Assistant Secretary of the Army for Civil Works was clearly preferential to the other, and that the background of shifting responsibilities within these two offices complicated the decision regarding an appropriate institutional home for the AGPR. The panel also felt that the decision about an institutional home may ultimately represent more of a policy decision than an analytical one, and that a final choice about institutional home would be more appropriately made by another body, likely the U.S. Congress. AGPR REVIEW OPTIONS In deciding upon the appropriate type of review for a Corps planning study, the Administrative Group for Project Review would choose among three options (Figure 6–1): Current Review Procedures. The current process would continue (which includes the District Office, Corps Headquarters, a review team in the Office of the Assistant Secretary of the Army for Civil Works, the

OCR for page 57
Review Procedures for Water Resources Project Planning FIGURE 6–1 Review options. *In some complex planning studies, it may be appropriate to initiate review during study reconnaissance.

OCR for page 57
Review Procedures for Water Resources Project Planning Office of Management and Budget, and, where applicable, other federal agencies such as the Environmental Protection Agency; see Chapters 2 and 3 for discussion of these various reviews within the current procedures). This option is likely to be applicable primarily to smaller, lower-cost, lower-risk, noncontroversial projects. Internal Review. An internal review process would be managed by the Administrative Group for Project Review, which would select the reviewers. The scale could be either small or large, differing by the number of panelists and the time allotted for review. A key distinction between the current process and internal review is that in the internal review process, a special review panel is appointed, and the panel may include experts from outside the Corps and it may also include Corps staff from other districts. Internal review panels should usually consist of a balance in the number of Corps of Engineers professional staff and non-Corps experts. External, Independent Review. The Administrative Group for Project Review would contract with an outside organization for the selection of review panelists. This type of review would be for large-scale, expensive, and/or controversial projects. External, independent review panels should consist of experts independent from the Corps, and the panel members should be selected by an organization independent of the Corps. The AGPR’s choice about the appropriate level of review should not be unilateral, however, and this decision should be open to review upon petition by interested parties. If the AGPR is located in the Office of the Chief of Engineers, appeals should be permitted through the Office of the Assistant Secretary of the Army for Civil Works. If the AGPR is located in the Office of the ASA(CW), appeals should be lodged with an executive-level body (e.g, the OMB or the Council on Environmental Quality). Moreover, the Administration—perhaps through a memo from the OMB director—should be able to request a review of a Corps planning study. Congress should also be able to request a review through a congressional resolution or other legislative action, and not simply through committee language accompanying a bill. The entire appeal process should extend no longer than 60 days. Internal Review Nominees for internal reviews may be selected from government, the private sector, or academia. Rules should be required to ensure an open

OCR for page 57
Review Procedures for Water Resources Project Planning nomination process. The names of the nominees and their nominators should be published prior to final appointment. The review panel should usually consist of a balance in the number of Corps professional staff and non-Corps experts. Final authority for the selection and appointment of reviewers would rest with the Administrative Group for Project Review. External, Independent Review As part of external, independent review, the Administrative Group for Project Review would contract with an outside organization to select an external and independent review panel. There are precedents for federal agencies having an external group manage their review processes. Examples include the NASA Life Sciences Branch (reviewed by the Universities Space Research Association), the U.S. Department of Army Medical Program (reviewed by the American Institute of Biological Sciences), and the U.S. Department of Energy’s Office of Science and Technology (reviewed by the American Society of Mechanical Engineers). Independent review of Corps planning studies could also be delegated to an independent federal oversight group, such as the Department of Energy’s Defense Nuclear Facilities Safety Board, which provides advice to ensure adequate protection of public health at nuclear facilities. Other options for organizations to oversee this independent review include the National Academies, the National Academy of Public Administration, and professional science and engineering societies. The Administrative Group for Project Review should assist external reviewers in understanding the Corps’ assumptions and methods in the relevant study. To assist a review panel, the AGPR should compile a summary document that clearly explains the contents, assumptions, models, and methods contained within a Corps planning study, project design, reoperations decision, or other relevant analyses. As noted earlier, preparation of this summary document will be a key responsibility of the AGPR. During an independent panel’s review, the Administrative Group for Project Review should be available to answer questions about a planning study’s evolution and contents, as should staff from the relevant Corps District Office(s). The AGPR should maintain communication between the Corps and the review panel through the course of the review, without compromising the review’s independence. Frequent communication will help the review panel understand the technical and practical implications of its recommendations. The AGPR should also promote cooperation and communication between the review panel and other parties with stakes and interests in the study at hand. The AGPR should typically

OCR for page 57
Review Procedures for Water Resources Project Planning provide administrative support for external review, although a consultant could be hired for this purpose. ENSURING EFFECTIVE AND CREDIBLE REVIEW Defining the Terms of Review An issue that frequently arises in review, and one not always easily agreed upon, is defining a review panel’s boundaries of inquiry. It is not uncommon for an agency or other administrative group to try to limit a review panel’s deliberations. For example, review panels may be charged to limit their discussions to “science” issues, and to not comment upon “policy” issues. Within the Corps, this issue has often been reflected as a tension between issues defined as “technical” and those defined as “policy.” However, the line between technical and policy issues is often blurred, and it is often difficult to clearly separate them. Review should be conducted to identify, explain, and comment upon assumptions that underlie economic, engineering, and environmental analyses, as well as to evaluate the soundness of models and planning methods. Panels should also be able to evaluate whether the interpretations of analysis and conclusions based on analysis are reasonable. To provide effective review, in terms of both usefulness of results and of credibility, review panels should be given the flexibility to bring important issues to the attention of decision makers. However, review panels should be instructed to not make a recommendation on whether a particular alternative should be implemented, as the Chief of Engineers is ultimately responsible for the final decision on a planning or reoperations study. Responding to Review To ensure effective reviews, it is important that there be a clear understanding from the outset of the objectives of the review and how the review’s results will be used (Kostoff, 2001). To help ensure that reviews do not become pro forma exercises, the primary client of reviews—the Chief of Engineers—should respond in writing to each key point in a review. The Chief should either agree with a point and explain how it will be incorporated into the planning or other study, or the Chief

OCR for page 57
Review Procedures for Water Resources Project Planning should reject the comment, providing an explanation of why the Corps is choosing to ignore it. A Review Advisory Board A Review Advisory Board (RAB), which should be a small, independent group that meets periodically to review the process and activities of the Administrative Group for Project Review, should be established. The Review Advisory Board would not perform study reviews. It would examine and advise upon the processes for selecting reviewers and establishing independent external review bodies. The Review Advisory Board would assess processes for ensuring consistency, thoroughness, and timeliness of reviews. It would consider both past studies and prospective studies and projects. As part of this responsibility, the Review Advisory Board also should periodically evaluate the scopes of review proposed by the Administrative Group for Project Review. This is important to help ensure that charges to review panels are clear and that the review scope includes appropriate technical and policy considerations. Finally, the Review Advisory Board should periodically review a sample of the summary documents produced by the Administrative Group for Project Review to ensure the clarity and comprehensiveness of those documents. The Review Advisory Board could suggest changes to enhance the quality of the review process. It would periodically issue reports to the office that houses the Administrative Group for Project Review. The Review Advisory Board would use background materials provided by the Administrative Group for Project Review, make site visits as necessary, and incorporate information from public comments when available. This panel discussed the prospects for the Review Advisory Board to enlist experienced and qualified water resources experts, and was concerned that a board mandated only to review review processes might be too narrowly structured to attract highly qualified scientists, engineers, and analysts. The RAB’s responsibilities may thus need to be made part of a body charged with a larger advisory mandate to the Corps.

OCR for page 57
Review Procedures for Water Resources Project Planning Timing and Administrative Costs of Review Timing Review can be beneficial at several stages in the planning process. The panel has incorporated specific steps into the planning study time line of Figure 3–1 to accommodate the review process (Figure 6–2). A preliminary determination would be made within three months regarding the possible need for, and scale of, review. If review is needed, the Administrative Group for Project Review would begin the process of identifying and selecting reviewers. When the Alternative Formulation Briefing (in which a Corps District Office releases a feasibility study’s alternatives for consideration) is later held, the Administrative Group for Project Review should reexamine its preliminary determination of the need for a review and the scope of the review. When either an internal or an external review is conducted, the review process should be initiated early in the study. The reason for this early start is that it is useful for Corps District-level planners to have evaluations from reviewers on the assumptions, methods, and data to be used in the feasibility study. If a review is not undertaken until after the Alternative Formulation Briefing, it may be too late to provide useful assessments to Corps District planners. The circumstances surrounding the Corps’ Upper Mississippi River-Illinois Waterway draft feasibility study (Box 1–1) provide a good example of the value of initiating review at an early stage. If a synopsis of the scoping process and proposed analytical techniques had been submitted for review early in that study, review could have provided input that might have averted questions raised later. Reviews of planning studies should generally begin at about the time the reconnaissance report is certified and the feasibility study begins. In the most complex planning studies, there may even be instances where review would be useful during the reconnaissance study. In these cases, however, the review panel should generally be disbanded after it conducts its evaluation to preclude the possibility of the panelists becoming defenders of their study. In some cases it may be desirable to defer initiation of the review process until the Alternative Formulation Briefing. The advantage is that the planning process would have progressed to a stage at which substantial material would be available for review. Problems and opportunities would have been identified, forecasts of future conditions would have been made, alternatives would have been formulated, and estimates of

OCR for page 57
Review Procedures for Water Resources Project Planning FIGURE 6–2 Corps water resources project planning process and review milestones. *In more complex planning studies, it may be appropriate to initiate review during study reconnaissance.

OCR for page 57
Review Procedures for Water Resources Project Planning benefits and costs would be available. A limitation of waiting until the Alternative Formulation Briefing is that substantial resources would already have been expended. In more complex and lengthy planning studies, reviews could be usefully conducted at different stages of the planning study. For example, a review panel could meet briefly in the early stages of a planning study to comment upon key assumptions and methods, then reconvene later during the planning study for a more comprehensive review (always bearing in mind the caveat about panels not becoming too beholden to the results of their review and not evolving into defenders of their recommendations). The Administrative Group for Project Review should make a final determination of the need for and scale of the review after reviewing public comments obtained during the Alternative Formulation Briefing and comments from the project manager and other interested parties. This panel envisions four possible outcomes: (1) the preliminary determination was correct and no changes are needed, (2) the preliminary determination suggests that no review is needed, but public comments suggest an internal review is needed, thereby necessitating establishment of an internal review panel, (3) the preliminary determination suggests that an internal review is needed, but public comments suggest an external review is needed, thereby requiring the Administrative Group for Project Review to contract with an outside organization for the selection of an independent review panel, and (4) the preliminary determination suggests that an external review is needed, but the status of the project and public comments suggest that an internal review is sufficient. A review would be performed in parallel with the preparation of the Feasibility Study and an Environmental Impact Statement (EIS), and it would be completed and reported prior to completion of the Feasibility Study and EIS. Results of the review (whether managed by the Administrative Group for Project Review as an internal review or done independently) may be provided to and used by the Corps team conducting the Feasibility Study and EIS, in an iterative way, to improve decision making. The issue of the appropriate timing of review raises questions regarding responses to review and roles of the review panel. When a review is conducted in a planning study’s early stages, the review should be submitted to the Corps’ District Engineer, who is the Corps official most directly in charge of the planning study. The District Engineer should prepare a written document explaining how the Corps intends to incorporate the review’s recommendations into its planning study.

OCR for page 57
Review Procedures for Water Resources Project Planning TABLE 6–1 Estimated Annual Costs of Review Entity Cost Item Estimated Cost AGPR 4 professionals, at $150k each $0.6M Support costs for each professional ,$50k each. $0.2M Subtotal $0.8M RAB 5 members   Support costs for meetings, including staff, travel and other compensation as appropriate $0.3M Subtotal $0.3M Review 6 experts per review   5 reviews per year, assuming $100k costs/review/year $0.5M Report publication and dissemination $0.4M TOTAL   $2.0M As noted previously, when reviews are initiated early during a planning study, and with panels that track the Corps’ responses to its reviews over time, there is a prospect that a panel would assume an advisory role and may also become beholden to and defensive of its views. It is important that a review panel focus on its specific tasks and not become defensive in the event that past recommendations are not implemented to a panel’s satisfaction. This may become a greater issue with lengthy planning or reoperations studies, the most complicated of which may require years to complete. One way in which this potential problem might be averted is to invoke different review panels at different stages of the study process. Another strategy that may improve review is to have panel members serve on multiple panels, as this would help standardize evaluation across multiple planning studies. Administrative Costs An issue related to the funding of review is how the sources of funding might affect review independence. It would be natural to expect the Corps of Engineers (as opposed to a higher-level oversight organization) to fund the reviews described in this report. But this may constrict the review’s independence somewhat, as even the most objective performers may be motivated to satisfy sponsors, possibly in hopes of maintaining future funding continuity. There are examples of independent organizations providing balanced and credible reviews for agencies that provide resources for the reviews, and this issue may not constitute a major con-

OCR for page 57
Review Procedures for Water Resources Project Planning cern, but it should nonetheless be considered when resources for review panels are provided. Regarding an estimate of review costs, it is difficult to provide a precise estimate, as the cost will ultimately be a function of the number of staff, the level of staff support, and the number of reviews and reviewers. But this panel derived a first approximation of the annual costs of maintaining an effective review process, arriving at a figure of roughly $2 million. Table 6–1 provides an approximate budget, which will vary depending on the number of reviews, panelists, and professional facilitators and on the types of reviews conducted (e.g., face-to-face meetings; videoconferencing; mail).