Risk Analysis and Uncertainty in Flood Damage Reduction Studies (2000)

The Committee on Risk-Based Analyses for Flood Damage Reduction embarked upon this study assuming that it would produce a technical report regarding the application of risk analysis within the Corps of Engineers 's flood damage reduction studies. The charge to the committee was:

The Secretary (Army) shall enter into an agreement with the National Academy of Sciences to conduct a study of the Corps of Engineers ' use of risk analysis for the evaluation of hydrology, hydraulics, and economics in flood damage reduction studies. The study shall include—

1. an evaluation of the impact of risk-based analysis on project formulation, project economic justification, and minimum engineering and safety standards; and

2. a review of studies conducted using risk-based analysis to determine —

   1. the scientific validity of applying risk-based analysis in these studies; and

   2. the impact of using risk-based analysis as it relates to current policy and procedures of the Corps of Engineers.

As the committee proceeded with its study and its discussions, it learned that the issues of risk analysis within the Corps were quite complicated, and that a full understanding of these methods and their applications required a broader investigation than the committee antici-

pated at the outset. To adequately address its charge, the committee ultimately examined several topics in addition to its technical analysis of risk-based techniques.

Thus, while this study presented several challenges, it also offered a unique opportunity to analyze a host of critical, emerging issues in statistical hydrology, geotechnical engineering, economics, communications, and public policy. In this study, these multiple disciplines, and the committee members representing them, were bound by common interests in risk analysis, floods, Corps of Engineers practices, and floodplain management.

The Corps of Engineers's adoption of risk analysis procedures that explicitly recognize and quantify hydrologic, hydraulic, geotechnical, and economic uncertainties should lead to projects better tailored to local conditions and to available information, thus better achieving social objectives and flood damage reduction goals. For example, the new procedures represent a significant advance over traditional levee freeboard requirements that led to inconsistent levels of flood protection among projects. The replacement of this long standing approach with new risk analysis methods required significant conceptual and methodological development. The new techniques are a significant step forward and the Corps should be greatly commended for embracing contemporary, but complicated, techniques and for departing from a traditional approach that has been overtaken by modern scientific advances. There should be no turning back from this important step forward.

It bears repeating that the former levee freeboard standard did not provide consistent levels of flood protection across the nation. A consistent protection standard must properly account for local and regional differences in topography, hydrology, and hydraulics, which the standard freeboard approach did not. For instance, as little as 2 feet of freeboard may be required to provide adequate flood protection in some areas, while in others, as much as 6 feet may be required. The traditional freeboard standard masks a significant degree of variation of risk of levee failure for citizens protected by these levees. This variation in risk of failure can be quantified by the Corps's new risk analysis procedure.

While the Corps is to be strongly commended, it is important that the Corps promptly address and resolve issues identified in this report in order for risk analysis methods to eventually become well founded, well documented, and correctly executed. Risk analysis methods need to be clearly understood and clearly documented, and results effectively com-

municated to the public, its elected officials, other federal agencies, the U.S. Congress, and to the engineering community.

The committee reviewed the Corps's risk-based applications, including the Hydrologic Engineering Center 's Flood Damage Assessment (HEC-FDA) computer program for calculating flood damage risk. The new method builds upon the deterministic approach to flood damage estimation that evolved over decades of use. The new method benefits from these decades of experience, but also suffers from the difficulty of translating deterministic practice into a probabilistic framework. While the Corps is to be praised for its adoption and implementation of the risk analysis methods, several technical issues remain only partially addressed. Particular concerns relate to models of hydrologic, hydraulic, and geotechnical uncertainties, performance metrics, and the economic uncertainty analysis.

According to the U.S. federal Principles and Guidelines, estimates of expected annual damages (EAD) are the primary criterion (performance metric) for project selection. These are supplemented by the annual exceedance probability (AEP), which describes the likelihood of flooding in areas that should be protected. These two project performance metrics are important and appropriately summarize economic and safety dimensions of system performance. They adequately characterize the performance of flood damage reduction projects.

For levee certification and engineering purposes, it is useful to calculate other system reliability measures, such as the conditional nonexceedance probability for the 1 per cent flood (this describes the uncertainty in the water height of the flood with a 0.01 probability of occurring in a given year). In the committee's view, however, such engineering reliability measures, which are difficult to understand, should not be used to communicate flood risk to the public. Conditional nonexceedance probabilities are an awkward combination of traditional and new risk-based concepts and are easily misunderstood. The concept of annual exceedance probability is clearer and provides the needed information. The committee thus recommends that the Corps use annual exceed-

ance probability as the performance measure of engineering risk.

The construction of risk measures rests upon complete and accurate specification of the uncertainties in each component of an analysis, and upon correct implementation of procedures to: (1) estimate knowledge uncertainties, (2) incorporate those uncertainties in the risk analysis, and (3) properly propagate the uncertainties in individual parts of the analysis to the final results. As the current method has shortcomings in these areas, the committee recommends that the Corps improve its analysis of economic, hydrologic, hydraulic, and geotechnical engineering uncertainties.

The committee reviewed the Corps's risk analysis model and the HEC-FDA computer code used to perform calculations. A concern is whether all of the important uncertainties are included in the analysis and if those uncertainties are appropriately represented. The Corps's method should clearly distinguish between natural variability and knowledge uncertainty. In some cases, such as stage-discharge relations, uncertainties due to natural variability appear to be incorrectly subsumed within knowledge uncertainties; in other cases, such as geotechnical levee performance, knowledge uncertainties are incorrectly subsumed within natural variability.

The committee thus recommends that the Corps focus greater attention on the probabilistic issues of identifying, estimating, and combining uncertainties. This recommendation is important because, in the way the analyses are conducted, knowledge uncertainties (lack of scientific understanding of events and processes) explicitly affect project performance measures, but natural variability does not.

The committee recommends that the Corps strive to reduce the considerable variation in the estimates of water surface elevation when using different models of river hydraulics. The Corps's experiences in applying alternative methods to estimate flood stage indicate that there can be substantial differences in the results.

The committee recommends that the Corps's risk analysis method evaluate the performance of a levee as a spatially distributed system. The geotechnical evaluation of a levee, which may be many miles long, should account for the potential for failure at any point along the levee during a flood. Multiple modes of failure (e.g., overtopping, embankment instability), correlation of embankment and foundation properties, and the potential for multiple levee section failures during a flood should also be considered. The current procedure treats a levee within each damage reach as independent and distinct from one reach to the next. Further, within a reach, the analysis focuses on the portion of

each levee that is most likely to fail. This does not provide a sufficient analysis of the performance of the entire levee. This has important implications for not only geotechnical and economic analysis of flood damages, but also for levee certification.

The Corps's new geotechnical reliability model would benefit greatly from field validation. The nation has many years of experience with levee performance and, unfortunately, also with levee failures. Much of this experience is documented and accessible to federal agencies. The committee recommends that the Corps undertake statistical ex post studies to compare predictions of geotechnical failure probabilities made by the reliability model against frequencies of actual levee failures during floods. The committee also recommends that the Corps conduct statistical ex post studies with respect to the performance of other flood damage reduction structures. These latter studies should be conducted in order to identify vulnerabilities (failure modes) of these systems and to verify engineering reliability models.

A key innovation of the Corps's new risk analysis procedures is the derivation of the uncertainty in economic benefits due to knowledge uncertainties. This analysis is performed using a Monte Carlo procedure that evaluates expected annual damages using different possible combinations for hydrology, hydraulics, geotechnical and economic model parameters. This uncertainty analysis is currently performed as if the errors for each individual structure that might suffer damage are independent of one another. This fails to include the correlations and interaction among these parameters and thus misrepresents the uncertainty in damages in individual reaches, and for the project as a whole. For example, if a house has a higher economic value than estimated, or is surveyed to be at too low an elevation, similar houses on the same block are likely to experience similar errors. Unfortunately, the current Corps procedures neglect these sources of correlation and as a result misrepresent the overall uncertainty in project damages and benefits.

For similar reasons, the Corps's procedure incorrectly computes the uncertainty associated with differences in benefits from alternative projects components. The current practice of summing and subtracting percentiles of uncertainty distributions for damages in different reaches, and for alternative project components, is statistically incorrect unless there is

perfect correlation among the values. This is not the case, although there are strong interrelationships. As a result, the uncertainty assigned to the computed annual project damages and benefits is incorrect.

The assessment of uncertainty in economic performance could be significantly simplified to yield more precise measures of uncertainty by focusing on uncertainty in project benefits for each individual structure, while considering the interrelationships among those uncertainties instead of accumulating damages over damage reaches. What is critical is not accurate assessment of flood damage, but rather accurate assessment of the degree to which the project will reduce flood damage. If the flood damage to each structure in the floodplain is estimated both with and without the project plan, the benefit provided by the plan to this structure is the amount of reduced damage. Errors in estimating flood damage arise from errors in flood discharge, water surface elevation and the structure's economic value. These errors affect the damages both with and without the plan in nearly the same way; thus, when the benefit is calculated, the effects of these errors may nearly be cancelled out. The uncertainty in estimation of project benefits accumulated over all structures could thus be significantly less than the uncertainty in estimating project damage. Moreover, the impact of correlation among the analysis errors may be similarly diminished. Detailed study of alternatives to the current method of accounting for uncertainty in economic performance is needed to identify more efficient and statistically robust procedures.

The committee recommends that the Corps calculate the risks associated with flooding, and the benefits of a flood damage reduction project, structure by structure, rather than conducting risk analysis on damage aggregated over groups of structures in damage reaches.

The committee recommends that the Corps adopt and use a consistent vocabulary for distinguishing among natural variability, knowledge uncertainty, and measures of system reliability. Although difficult to assess, the distinction between natural variability and knowledge uncertainty is important because of the different roles these sources of uncertainty play in risk analysis.

In the early 1990s the Corps adopted a risk analysis approach to replace the practice of certifying levees that had 3 feet of freeboard above the 100-year flood. This former criterion provided different levels of flood protection to different sites. Both the new risk analysis approach and the levels of flood protection it provided were controversial. Negotiations with the Federal Emergency Management Agency (FEMA) led to the current practice of certifying a levee based on a three-tiered decision rule, using either 3 feet of freeboard, or a 90 percent conditional nonexceedance probability of passing a 100-year flood, or a 95 percent conditional nonexceedance probability. This criterion is a step forward and a reasonable transition from the older levee certification criterion into the new risk analysis framework. It has problems, however: (1) it still leads to different levels of flood protection for different projects, (2) the three-tiered decision rule is unnecessarily complicated, (3) the method evaluates levees individually and not as a system, and (4) certification considers only the 100-year flood, not the full range of floods.

The committee recommends that the federal levee certification program focus not upon some level of assurance of passing the 100-year flood, but rather upon “annual exceedance probability.” This is the probability that an area protected by a levee system will be flooded by any potential flood. This annual exceedance probability of flooding should include uncertainties derived from both natural variability and knowledge uncertainty.

The adoption and implementation of annual exceedance probability as the key criterion in levee certification will require significant resources and time. Until the measure of annual exceedance probability approach is adopted as the key criterion for levee certification, the committee recommends that the Corps and FEMA adopt a single conditional nonexceedance probability for levee certification.

The former certification criterion was flawed in that it produced vastly different levels of flood protection for different communities. The committee recommends that the certification criterion provide a uniform level of flood protection. Which level of protection to choose is not obvious. Insisting on the highest level of protection would mean that only a very small proportion of levees would be certified. In the committee's judgment, the certification criterion should be the level of protection provided to most people in the past—the median level historically provided. Based upon a small sample of all Corps flood damage reduction projects, the committee found that the median annual ex-

ceedance probability of Corps flood damage reduction projects is approximately 1/230.

This is the committee's best estimate of the median annual exceedance probability. To obtain a more reliable measure of the median annual exceedance probability of approved projects, the committee recommends that the Corps examine a larger number of flood damage reduction projects and audit the process of estimating the annual exceedance probability for these projects.

The committee also recommends that the Corps develop a table showing percentiles of variability in the annual exceedance probability of its flood damage reduction projects. By choosing an appropriate percentile value in this range, a corresponding level of assurance can be obtained that the expected level of protection is at least 100 years, as required. The lack of allowance for this variability led to the abandonment of the annual exceedance probability criterion during the 1990s.

Neither the U.S. Congress nor the Corps of Engineers have identified an explicit goal for management of the nation's floodplains. The committee is of the view that the goal of floodplain management should be to use the land for the greatest social benefit. Broadening the scope of Corps risk analysis and expanding the types of alternatives considered would provide more useful insight about how to best achieve this goal.

If conducted using the best information available, risk analysis can provide substantial insight for making floodplain management decisions. The Corps's risk analysis method quantitatively considers only direct damage reductions and costs of alternatives. It does not consider potential loss of life or environmental and social consequences. The benefits that can be claimed within Corps water resources project planning studies are specified by the federal Principles and Guidelines and not by the Corps.

To ensure that the Corps's flood damage reduction projects provide adequate social and environmental benefits, the committee recommends that potential loss of life, other social consequences, and environmental consequences be explicitly addressed in the Corps's risk analysis. This will improve the ability to make informed decisions about floodplain management and should ultimately lead to better use of our nation's floodplains. Floodplain management alternatives should not be limited to structural alternatives such as levees, dikes, and dams. In-

formation and communication alternatives, such as warning systems and zoning regulations, should be considered both separately and in conjunction with structural alternatives. The committee recognizes that the Corps alone cannot implement these recommendations, nor is such a broadening of the risk analyses likely to occur over a short period of time.

To appropriately include such consequences and their relative importance, the committee recommends that the ecological, health, and other social effects of Corps flood damage reduction studies, and the tradeoffs between them, be quantified to the extent possible and included in the National Economic Development Plan. More explicit efforts at including these types of consequences and values in the Corps's benefit –cost calculations should increase social benefits of the Corps's flood damage reduction studies. Examples of these consequences that are not included in the current benefit–cost guidelines contained within the Principles and Guidelines include lives saved (by structural and nonstructural projects), damages avoided to structures in floodplain evacuation projects, and preservation of biodiversity. Appropriate revisions of existing legislation, consistent with this recommendation, may have to be enacted by the U.S. Congress. The Corps should seek guidance from the Office of Management and Budget and seek consistency with other federal agencies on the use of alternative metrics for incorporating potential loss of life, environmental impacts, and other effects of floods.

The U.S. Army Corps of Engineers is a national and international leader in addressing flood-related problems. The committee applauds the agency's commitment and applauds Corps personnel in adopting, developing, and implementing the risk analysis approach. It is imperative that the Corps take the time needed to complete this new approach so that it achieves its potential. This new approach represents a significant step forward for the Corps, and for flood damage reduction studies, in general.

The committee commends the Corps for recognizing the limitations of the former procedures and for having the courage to undertake the development of a new, controversial, and technically difficult risk analysis approach. We offer this report in the spirit of constructive advice and in hopes of promoting wiser use of the nation's floodplains.