Review of the Prince William Sound, Alaska, Risk Assessment Study (1998)

After the Exxon Valdez accident in Prince William Sound (PWS), Alaska, in 1989, many suggestions were made as to how to prevent accidents in the future. A combination of shippers, local groups, and the U.S. Coast Guard contracted for a study to be done by a team from Rensselaer Polytechnic Institute, George Washington University, and a Norwegian classification society, Det Norske Veritas. Using data from PWS, the study team developed three models to examine the current level of risk and to evaluate proposed risk mitigation measures to reduce the amount of oil spilled in the sound. The initiators of the study asked the National Research Council (NRC) to provide a peer review of the PWS risk assessment. This report authored by the NRC Committee on Risk Assessment and Management of Marine Systems, is that review.

The Prince William Sound, Alaska, Risk Assessment Study (PWS Study) is an important step forward in using probabilistic risk assessment methods to assess the safety of transporting oil in large tankers in PWS. Strengths of the study include attempts to use probabilistic methods at the basic modeling level (fault tree logic diagrams and the marine accident risk calculation system (MARCS); searches for data (available databases, reasonably well designed questionnaires, and an attempt to involve outside experts); presentation of the results in a variety of forms (accident frequencies and oil outflow probabilities); and involvement by stakeholders.

Despite the advances in marine risk analysis in the PWS Study, the NRC committee is quite critical. The PWS Study can only be understood or, more importantly, evaluated in conjunction with two large volumes of supporting documents, called the Technical Documentation, and even then some things are not clear. The most significant weaknesses of the PWS Study are: (1) the lack of an overarching framework to ensure the consistency and logic of the analyses; (2) the lack of a clear description of how the models were implemented, the probabilities calculated, and the results reached; (3) the inaccessibility of the proprietary data on which the results are based; (4) the treatment of human and organiza-

tional error; and (5) the appearance that conclusions are precise and logical, when in fact, they are neither.

The PWS Study was an ambitious effort to combine several modeling approaches and site-specific data with international data to estimate risks and recommend measures for mitigating risks. In addition, the study approach involved close and continuous interaction with a nongovernmental citizens’ group, the PWS Regional Citizens’ Advisory Council (RCAC), which represents most of the stakeholders in the region. Thus, the PWS Study is less an independent analysis of risk than a mutually agreed upon description of issues and recommendations for mitigating risk. This interaction had both positive and negative influences on the study.

Risk analyses promise to improve the safety of oil transport in many congested shipping areas in ports, straits, and sounds worldwide. However, the modeling approaches used in the PWS Study cannot be directly transferred to other areas. The PWS Study should be considered as a first step in marine systems risk assessment. Even in this regard, it should be viewed as a preliminary first step. The study was extremely ambitious, involving two major contractors and a consensus approach with the RCAC. A limiting feature of the PWS Study was that the events being assessed were rare: the database of actual accidents was sparse, consisting of one grounding and one ice collision. Because the data were very limited, the analytic results and the resulting conclusions are not robust and are necessarily uncertain.

The following discussion is a summary of the conclusions of the NRC committee’s review of the PWS Sudy concerning its models, including the use of data, the treatment of human factors, the risk reduction measures analyzed in the study, and the applicability of this study to other locations.

The PWS Study used three modeling approaches: MARCS (marine accident risk assessment system), fault trees, and simulation. A fourth model for estimating the volume of oil spills was used in conjunction with each. The potential weaknesses of the MARCS approach, as used in the PWS Study, are listed below:

• the lack of dynamic modeling

• the assumption that all ships traveled at an average speed

• the assumption that all ships adhered to the collision avoidance rules (i.e., that there were no “rogue” ships)

• the exclusion of human factors

• the exclusion from the powered grounding model of accidents caused by failures to make required course changes

Although the fault trees in the PWS Study appear to be reasonably complete, they were not developed or based on basic event data. The top frequency in the event tree, which was distributed to the basic events level, was developed from expert judgments. (The experts were three employees of Det Norske Veritas, a Norwegian classification society, who had substantial maritime experience.) However, this is not a true fault tree model

because no logical analyses relating basic events to accidents were included. The committee recognizes that data were not available at the basic events level, but it might have been better if the study had used only the top blocks (inputs to the top event gate) and labeled the estimates expert judgments instead of implying that the analysis was based on actual fault trees. In addition, the estimates do not include uncertainty ranges; thus they give a false sense of completeness. This approach could result in risks being underestimated and risk reduction measures being overlooked.

The simulation model in the PWS Study appears to be relatively straightforward and could handle the large number of fishing boats and the smaller number of other vessels. At the heart of the simulation were the probabilities of incidents and the conditional probabilities of accidents. Most parameters in the conditional probabilities were based on expert judgments encoded through relatively long (up to two hours) questionnaires given to 162 people, including pilots and tanker officers. The NRC committee has many concerns about the use of these questionnaires.

Many assumptions about data and operations were used in the analyses and in the assertions about the analyses. The PWS Study team stated that the assumptions were coordinated with, and agreed to by, the PWS Steering Committee.¹ The agreement of the steering committee does not provide scientific credibility. At best, these assumptions are poorly explained and are not supported by the study. In general, the impact of the assumptions was not explored in the PWS Study.

The PWS Study asserts that the relative closeness of the numerical results of the three models indicates the correctness of the results and the validity of the analytic methods. However, the methods, as implemented, were all based on the same input data and modeling assumptions. Therefore, a reasonable case can be made that the results were bound to be comparable. The NRC committee does not find the report’s argument compelling.

Sensitivity analyses and discussions of uncertainty were not included in the PWS Study, with the exception of a discussion of the uncertainty in the fault tree modeling of powered grounding in the Narrows. No analysis presented in the study enables the reader to understand the effect of uncertainties and assumptions on the results.

Large scale models are critically dependent on the proper treatment of all relevant and available information, including accident statistics, weather data, operational data, and carefully encoded expert opinion, when necessary. In spite of the PWS Study team’s diligent efforts to collect applicable data, the available environmental data were sparse; operational data also had to be supplemented with worldwide data. The NRC committee questions the applicability of some of the supplemental data.

Human factors must be a critical part of risk assessments, especially for crew-oriented marine systems. Based on related substantial work in the areas of aviation safety and

nuclear power plant safety, the NRC committee has concerns about how human factors were treated in the PWS Study, particularly with regard to the use of expert judgments. The committee appreciates that insufficient data were available, but is concerned with the way expert judgment was used to infer the probabilities of incidents and accidents attributable to structural, mechanical, or human errors.

Because of the lack of essential and objective data, the PWS Study team found it necessary to elicit and analyze expert judgments to complete their models. The study team appears to have tried to take into consideration and make adjustments for experts’ fatigue, levels of understanding, individual scale bias, and variability in responses to the questionnaires, but the NRC committee has fundamental questions about expert community bias or viewpoint, as well as residual questions about the consistency of the responses and their application. The application of sophisticated statistical techniques to the responses to the questionnaires tends to mask these problems. Further complications arise from the use of a subjective “worst case” approach. Mixing worst cases with probabilities makes interpreting results extremely difficult.

The PWS Steering Committee was involved regularly with the risk assessment study, but it is not clear if the recommended risk reduction measures were modified at the request of the steering committee. If modifications were made, they may have been entirely appropriate, but the development of the list of risk reduction measures is not clearly explained. It is clear that some of the proposed risk reduction measures were not included on the final list.

The PWS Study concludes that the most effective measure for reducing accidents is revising traffic rules and that the most effective measures for reducing oil outflow are improving human and organizational performance. The NRC committee agrees that the conclusion about traffic rules follows from the data and the model. However, the committee found many weaknesses in the probabilities of human error, including a lack of reliable data and necessarily arbitrary assumptions incorporating human error into the models. Therefore, the committee has little confidence in the study conclusion regarding measures of human performance.

The recommendations by the study team were made with an eye toward the overall limitations of their methods and were formulated to be as specific as possible without recommending specific solutions. The recommendations were based on a representative year, 1995, but no attempt was made to determine how representative that year was or whether a fictitious year could have been developed that would have been representative. If the overall objective was simply to reduce risk, the use of a representative year hardly matters. However, if the objective was to rank relative risks, comparisons with 1995 might result in distortions. If the objective was to rationalize and justify a risk reduction strategy, using a representative year may be cause for concern. For example, one could conclude that no additional risk reduction measures should be undertaken. The PWS Study provides no guidelines to measure the effectiveness of specific risk reduction measures.

The conclusions and recommendations of the PWS Study are location-dependent and cannot be assumed to apply to other regions. Careful consideration should be given to the extent to which the situations under examination are similar to the ones analyzed in the PWS Study, from the broad features of the operating environment to the handling of individual variables that appear to influence the result. Because relatively little sensitivity analysis was done on the models used in the PWS Study, the effects of individual variables on study results are not easy to discern or to apply elsewhere.

Replication of the modeling approach also depends on collecting and analyzing similar types of location-specific data. Some of the data in the PWS Study are proprietary and depend on the cooperation of the firms that own it, and some depend on high level access to local experts. Other data, such as information on vessel tracks derived from the vessel identification capabilities of the PWS vessel traffic system, may be relatively difficult, if not impossible, to obtain for other ports.

In short, other regions of the country should not try to replicate the PWS Study. The approach used in the PWS Study has three elements: the analytical methods (George Washington University dynamic modeling, MARCS, and fault trees), the manner of implementing the methods, and the results. The methods are clearly and obviously applicable to other areas, with varying degrees of effectiveness and difficulty. The manner of implementation may be applicable depending on the circumstances; but the details are clearly not applicable. The results are not applicable except as a list of topics for consideration. A determination of unacceptable risks and acceptable risk reduction measures are always greatly influenced by local considerations, and local communities should participate actively in the early stages of a study to identify the objectives. The analysis should identify the factors and situations that tend to increase or decrease risk and the effectiveness of particular measures in reducing risk when system-wide effects are taken into account.

The PWS Risk Assessment Study is an important step forward in the use of probabilistic risk assessment methods for assessing the safety of transporting oil in large tankers in PWS. Strengths of the study include: attempts to use probabilistic methods at the basic modeling level; searches for data from available databases and well designed questionnaires; the presentation of results in a variety of forms; and stakeholder involvement. However, the PWS Study fails to measure up to the review criteria adopted by the NRC committee, which incorporate and expand upon criteria used for all NRC reviews. In particular, as applied in the PWS Study, the weakness of the analytic methods raises questions about whether the results represent a scientifically-based assessment of risk. Close involvement of the stakeholders throughout the study process also undermines the independence of the study and could compromise the scientific validity of the study results. Although the PWS Study provides valuable information to those in decision-making roles, it does not have the scientific rigor necessary to compare alternative measures. Taking into account stakeholder involvement, the limited data used for the modeling, and the lack of

transparency in the modeling, the NRC committee concludes that the PWS Study is a first step in the right direction but that applying it directly to other areas will require major changes in methods and procedures.

Changes that would make it more generally applicable include the following:

• providing an overarching study framework

• expanding the consideration of human factors

• disclosing the underlying data

• analyzing sensitivities and uncertainties

A major improvement would be to ensure the study team’s independence of the steering committee, which should only be involved with establishing and monitoring specific goals and objectives and facilitating the collection of information.