National Academies Press: OpenBook

Simulated Voyages: Using Simulation Technology to Train and License Mariners (1996)

Chapter: 7 Simulation and Simulator Validity and Validation

« Previous: 6 Simulator-Based Training and Sea-Time Equivalency
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

7
Simulation and Simulator Validity and Validation

The levels of trajectory accuracy and fidelity needed and delivered in the replication of ship maneuvering behavior for simulation training in computer-based and manned-model simulation are debated within the hydrodynamic modeling, marine simulation, and marine education and training communities. Ship-bridge simulators are used for all types of operational scenarios. Whether all of the appropriate vessel maneuverability cues are present in the simulation or correctly portrayed, whether the trajectory of the ship is actually correct, and the relative importance of accuracy in these areas are all important issues in the use of computer-based marine simulation (NRC, 1992).

Ship-bridge simulators are not only developed independently of the vessels that they simulate, they are routinely used to permit training in multiple hull forms and sizes. As a result, some simulator facilities use either a number of models to meet the specific application needs of training sponsors or adjust their models to simulate a different type or size vessel. If these adjustments are not correct, the resulting trajectory predictions will be inaccurate, regardless of the quality of the algorithms used to generate them or the apparent validity of the simulation. For these reasons, it is appropriate to validate each trajectory prediction model or perturbation to determine the capabilities and limitations of the product being delivered to the trainer, marine licensing authority, and licensing examiners and assessors.

The accuracy and fidelity of available ship-bridge simulators can vary significantly among facilities. These variations may result from differences in the mathematical models used to develop the scenarios or model modifications made by simulator facility staffs.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

THE FIDELITY-ACCURACY RELATIONSHIP

As mentioned earlier, fidelity refers to the realism or degree of similarity between the training situation and the operational situation being simulated. The two basic measures of fidelity are physical and functional characteristics of the training situation (Hays and Singer, 1989). In the case of a manned model, the model contributes to both. In the case of computer-based simulation, the mathematical model contributes to functional characteristics. Fidelity is determined subjectively. The level of fidelity required is determined by the training objectives, which, in turn, are based on task needs and training analysis (Chapter 3; Hays and Singer, 1989). Determinant measures may be used to aid in assessing the level of fidelity in a given simulation. Accuracy is inherently a determinant measure of how close something is to being exact. The accuracy of a trajectory prediction model is determined by measuring variations of the predicted trajectory with the actual trajectory.

Correlating Realism and Accuracy

In many respects, fidelity is more difficult to address than accuracy because it involves a subjective assessment of how real the simulation is. Balancing accuracy of trajectory modeling with fidelity of motion in visual scenes, for example, is very challenging. It is possible to provide a believable simulation using a simple trajectory model that, with a few minor validating adjustments, can appear to be realistic to be realistic to pilots and mariners in the specific harbor/ship situation. Yet, performing slightly different maneuvers than those used for validation can result in quite inaccurate trajectories. Indeed, all models have limited accuracy in various modes of which the trainer may be unaware. In general, this issue has not been addressed by simulation providers except to try to use the most accurate modeling approach economically available.

The accuracy of trajectory prediction models available to drive a simulation can be compared with the level of fidelity specified by the training analysis as necessary to achieve training objectives. The accuracy of trajectory prediction, for instance, is less important in courses where vessel maneuvering behavior is not an instructional objective than in courses where maneuvering is required to achieve the goal of certain learning situations or is the primary instructional objective.

Deliberate Departure from Realism

It is sometimes possible to enhance training effectiveness by departing from realism. As a general rule, in marine simulation, departures from realism are driven by limitations in training resources, rather than a conscious attempt to optimize training effectiveness. The most notable exception is the initial development

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

of manned models, a development borne out of practical necessity to safely train the prospective masters of very large crude carriers in shiphandling. The scaling inherent in manned models is believed by many to enhance training effectiveness, although there are questions regarding the effect of scaling factors on individuals who do not have a well-established frame of reference in the operation of ships of the categories being simulated.

Because computer-based simulations rely primarily on software-based mathematical algorithms, there is considerable flexibility that could be used to deliberately depart from realism. In marine simulation, however, the opposite approach has been the rule. To build and improve confidence among mariners, training sponsors, and marine licensing authorities, there are strong pressures to use the highest level of realism possible. Nevertheless, it is possible to alter the mathematical trajectory prediction models to accentuate certain vessel maneuvering behavior, for example, as an instructional technique to assist a trainee in becoming aware of this behavior. As a rule, such an approach is problematic, because it appears that only a few ship-bridge simulation staffs have the level of sophistication in instructional design and hydrodynamic modeling to effectively stage and control deliberate departures from realism.

A major technical consideration in the application of simulators and simulations is the need for consistently reproducible results from simulation exercises. Currently, there are no standards for the development, operation, or modification of simulators or simulations. As their use is expanded from training to performance evaluation, licensing assessment, and substitution of training for sea time, consideration needs to be given to the establishment of industrywide criteria and standards.

PHYSICAL AND MATHEMATICAL SIMULATION MODELS

From the modeler's perspective, the simulation user must specify the accuracy needed for particular training or licensing objectives. The simulation modeler (for physical models or computer-based models) can then assess whether that accuracy can be provided. Pilots, for instance, need very accurate models to properly portray bank effects and other complex interactions, whereas a less robust model may suffice to introduce very basic operational concepts and procedures to beginners. A detailed discussion of the hydrodynamic, physical models, mathematical models, and research needs in these areas is included in Appendix D.

Ship-Bridge Simulator Models

Current ship-bridge simulations are based on mathematical models derived by extrapolating hydrodynamic coefficients from towing-tank tests for a restricted

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

BOX 7-1 Anchoring Evolutions: An Example of Needed Research

During anchoring maneuvers, the capability to determine changes in speed over the ground is very important. Anchoring evolutions have proven very difficult to simulate realistically in computer-based simulators. With respect to hydrodynamics, deceleration needs to be realistic under the influence of an astern bell. The retarding effect of dredging an anchor (that is, maneuvering about an anchor dragged along the bottom to retard the movement of the bow) needs to be correlated with propulsion and environmental conditions. The anchor needs to have the effect of holding the ship, dragging, and holding the bow as the ship swings into the wind. Understanding these effects and the ability to model them effectively is a weakness in the current state of practice of mathematical modeling and varies considerably among the various mathematical models.

set of hull shapes. It is generally assumed that the database is technically advanced enough to give accurate simulations of ship maneuvers in deep and unrestricted waters.

The simulation of ship dynamics for shallow-water effects, anchoring evolutions, and ship-to-ship interactions are less technically advanced. Simulations involving these factors are generally less accurate. The simulation of vessels being towed is also limited by an absence of systematic test data.

Numerous mathematical models are currently used to drive the various simulators. Hydrodynamic coefficients used in the models cannot be easily exchanged among most facilities. If one simulator facility has a need for a model of a ship it does not have, acquiring that model can be time-consuming and expensive. A standard method for exchanging models or modeling coefficients would facilitate sharing of important technical information. Development of model modules is also needed so that modules can be validated more effectively and an individual module can be replaced with one representing different vessel characteristics (e.g., propeller, rudder, hull trim, draft) without requiring elaborate modifications.

Inconsistencies in modeling capabilities can arise from several causes, including the lack of appropriate data and a lack of access to existing data. Additional research is needed to extend databases in areas such as ship maneuvering coefficients and anchoring evolutions (see Box 7-1). The International Maritime Organization (IMO) has set some criteria for collecting full-scale ship maneuverability data and has established a five-year collection effort. The IMO's intent is to revise requirements for the data based on knowledge gained in the process.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

Applying computational methods to the determination of pertinent hydrodynamic parameters offers great promise as a solution to many current modeling problems. Practical developments and use in ship-bridge simulators cannot be implemented until existing research codes are applied to the specific computational tasks of maneuvering simulation. Two different levels of implementation might be possible.

At the first level, computational methods can be used to complement towing-tank testing in the determination of hydrodynamic coefficients. Examples where computational methods can play a useful role include ship-to-ship interactions in restricted channels and the use of more-elaborate computational methods that account for viscous effects to consider ship operations with small underkeel clearance in shallow water.

At the second level, computations of pertinent hydrodynamic effects could be performed with sufficient accuracy and speed to replace the traditional approach, which is based on curve fitting. This approach could be particularly useful for ship interactions and restricted water effects, where existing simulator models are severely limited. Based on three-dimensional panel methods, it is now possible to compute the relevant interaction forces and moments during the simulation, based on actual ship trajectories and channel topographies, thus avoiding uncertainties and limitations.

Full-scale real-ship experiments would advance the state of practice in modeling, particularly for shallow water and restricted waters with banks. Modeling of operations at slow speed and with reversing propeller situations also needs improvement. Vessels currently part of the U.S. Maritime Administration's Ready Reserve Fleet and some vessels in the Navy's Military Sealift Command fleet represent a possible source for data to validate and improve mathematical models.

A second source of information to improve simulation design could be the data bank of proprietary towing-tank information that exists at towing-tank facilities in the United States. Much of this data was developed for private clients and is currently unavailable for public use. If procedures could be developed to allow the public access to existing proprietary data without disclosing the source, simulation design might be significantly improved with a minimum investment.

Manned Models

Physical models provide a simple approach to training in the understanding and application of the hydrodynamics of ship motion in deep water and close-in operations, including docking and coming alongside, maneuvering in shallow water and near banks, and in ship-to-ship operations. The scaling of these ship dynamics, however, results in propulsion and rudder-action modeling inaccuracies. Also, because of the way human eyes spread when relating to the scale of the model, the trainee's stereoscopic vision is more acute on a manned model

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

than on the the ship. Currents, wind, and waves are also difficult to model and are usually done only to a limited extent, and then not all facilities.

Simulation Software Issues

As with any large computer program, the software developed for ship simulation is subject to potential limitations. These include:

  • errors or restrictions in the assumed hydrodynamic, model
  • numerical errors due to the reduction of the model, to computational form (such as time steps that are too large), and
  • programming errors (bugs).

Over time, age and insufficient maintenance may effect a program's relative quality and relevance, unless special efforts are made to provide updated versions.

Computer programs developed for ship-bridge simulators require substantial investments of expertise and effort, and this investment is normally protected by licensing agreements. Further protection is often achieved by distributing the code in an executable form that cannot be modified or transferred to different computational environments. Public-domain software is preferable from a user's standpoint and offers the significant advantage that it can be shared within the simulation and hydrodynamic communities to hasten the exchange of ideas, correct errors, and improve hydrodynamic models. This concept of ''open" software is particularly suitable if use of simulation becomes mandated (i.e., in licensing requirements). Access to "open" software would also facilitate validation of simulators and simulations.

CURRENT PRACTICE IN VALIDATION

As noted in Chapter 2, commercial air carrier simulators are evaluated and validated at a particular level, depending on their application, for a range of operating conditions. Generally, a commercial air carrier simulator represents a particular aircraft model with a cockpit arrangement specified by the operator. Air carrier simulators are re-evaluated every four months through the National Simulator Evaluation Program. Validations include both objective and subjective elements. The simulator's performance and handling qualities are evaluated according to engineering specification; pilot acceptance is determined through subjective validation.

Once the simulator has been validated, any changes in the performance characteristics necessitate revalidation. This approach is possible because simulators, which are generally developed concurrently with the airframe, are platform-specific and benefit from airframe development. (Marine simulators, which generally simulate generic ships and bridge equipment arrangement, are not ship-specific.)

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

Under present U.S. Coast Guard course-approval practices, formal validation of simulators and simulations is left to facility operators. There is no industrywide standard validation methodology. Typically, validation begins with the manufacturer of the simulator and its associated devices, hardware, and software. The manufacturer's performance evaluation is based on the manufacturer's internally generated criteria. Technical deviations (e.g., pixel size or color abnormalities or processing speeds) are adjusted to contract specifications. Because no specific vessel is usually being replicated and there are no international or fixed standards for bridge and engine room layouts, the location of navigation or communication hardware (e.g., radiotelephone, radar, automatic radar plotting aids, chart stand) are based on facility layout and management preference.

In general, simulator facilities qualitatively validate their simulators and simulations with respect to ship maneuverability and visual scenes and their integration and correlation. Evaluations of the accuracy and fidelity of the simulation image and vessel response characteristics are generally accomplished through subjective mariner testing. Several mariners with experience on ship types similar to those being simulated usually conduct their evaluation over a period of three to five days. During that time they are familiarized with the simulator and asked to evaluate the performance of the ship type under a variety of conditions. These conditions range from normal day and night operations, to abnormal, unusual, and emergency situations. The "test mariners" put the simulator through various maneuvers to determine the accuracy of the simulated maneuvering behavior.

Following each simulation run, manufacturer and facility personnel debrief the mariners, asking for descriptions of accuracies and inaccuracies in the simulated vessel's performance. The mariners may suggest changes in such factors as turn rate, response to rudder and engine commands, acceleration and deceleration, or visual scene relevance and accuracy. Facility operations personnel then make changes in the vessel or port database. This process continues on an iterative basis until the test mariners are satisfied, or until the operations personnel indicate that no more changes can be accommodated. This iterative model validation process represents the state of practice in the industry, but it is not endorsed by the committee.

FACILITY-GENERATED MODELS AND MODIFICATIONS

Development of Facility-Generated Models

Development of models by a facility's staff is not uncommon. They are often developed from modeling routines available in some ship-bridge software packages. The facility staff will develop its own models if they do not have one available in the software library acquired with the simulator or if the available model did not perform to the satisfaction of the trainees or the instructional staff.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

Some individuals at facilities are self-taught modelers who create and then validate their own models without the benefit of an outside perspective to ensure overall reasonableness and accuracy of the simulated ship's maneuvering behavior. In addition, students are sometimes asked for suggestions, which can result in model modifications.

Field Adjustments to Mathematical Models

Typically, facility operators continue to modify simulators and simulations after the initial manufacturer and test mariner validations are complete. In some instances, the process of refining and modifying the model can continue indefinitely. There may be several reasons why a facility operator makes field adjustments in the mathematical models.

First, in using the model there may be anecdotal indications that the accuracy and fidelity of the maneuvering behavior are different from those needed to achieve training objectives. In some cases, staff or others associated with a simulator may "correct" real or perceived deficiencies in simulations of ship maneuvers by modifying the hydrodynamic coefficients. Because the mathematical models used to represent the maneuvers include many terms and coefficients, the danger exists that ad hoc modification of one or several coefficients to improve a particular simulated trajectory will actually degrade others. Such modifications are often based on the perceptions of a small number of users, and their judgments may not be accurate.

A second reason a facility operator may adjust a model is if the simulator manufacturer does not provide support for model corrections or if the support is not timely. In one example, a facility was dissatisfied with the performance of the library model supplied by the simulator manufacturer for a unique vessel category. The simulator manufacturer's arrangements for model refinement did not provide for a timely resolution. As a result, the facility generated its own model using a software routine that was a component of software provided with their simulator.

A large number of mariners were exposed to the facility-generated model and asked to evaluate its performance. The coefficients were then adjusted to generate performance. The coefficients were then adjusted to generate performance they found acceptable. The facility reported that the resulting model produced more accurate vessel motions than the library model. Although mariner confidence was established in the simulation, the altered model was not reviewed by a hydrodynamicist to ensure its hydrodynamic accuracy and fidelity.

AN APPROACH TO SIMULATOR AND SIMULATION VALIDATION

At present there are no industrywide standards for simulators or simulations and no standard ship models; nor are there standard simulation scenarios for performance evaluation or licensing assessment. Without industrywide standards

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

there can be no assurance that training or licensing assessment goals and objectives are being consistently met at different facilities and on different platforms.

Continually modifying the model—with the lack of consistency that results—could be problematic for students training on simulators because each successive class may be conducted with a slightly or even dramatically different simulation than an earlier class. The level of model accuracy required and the extent to which simulation accuracy needs to be validated depends on the proposed use of the simulation. The practice of continually modifying models and the resulting inconsistency will become a major concern if the simulations are used for formal performance evaluations, licensing assessments, or training that results in remission of sea time.

For validating models at the U.S. Merchant Marine Academy, the institution's simulator specifications contain the following:

A wheelhouse poster containing general particulars and detailed information describing the maneuvering characteristics of each own ship modeled. This wheelhouse poster shall meet the requirements of IMO Res. A.601 (15). In addition a maneuvering booklet shall be provided for each own ship modeled. This booklet should include comprehensive details of the ship's maneuvering characteristics and other relevant data. It shall meet the requirements of IMO Res. A.601(15).

Standards for Training Simulations

Generally, mariner instructors believe that cadet training, rules-of-the-road training, and bridge team and bridge resource management training do not require high levels of accuracy, only behavior that is qualitatively correct. To teach basic shiphandling in deep-water operating conditions, moderately accurate ship hydrodynamic models may be adequate.

High accuracy and fidelity in hydrodynamic, channel, and harbor models are important for shiphandling training for conditions of shallow water, confined water, and small underkeel clearances. High fidelity is generally required for marine pilots who perform at a much higher level of detail and precision in confined waterways than do ships' officers in general. Thus, in the context of pilotage training, and for specific ship and port operations, both the ship and channel-environmental models need to be as accurate as possible. Pilots will reject simulations they do not believe to be accurate.

Standards for Performance Evaluation or Licensing Assessment Simulation

For effective mariner performance evaluation or licensing assessment on a simulator, industrywide standards need to be established. To ensure consistency

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

in scenarios used for evaluation and assessment, standards need to be developed based on carefully defined levels of simulation validity that is required or is acceptable for the appropriate level of evaluation or license being tested. Not all evaluations or licensing assessments require the high face or apparent validity possible with a full-mission ship-bridge simulator. Insofar as practical, the process should require the use of standardized mathematical ship models and operational scenarios and databases to ensure consistency.

To the degree that ship maneuverability affects individual performance during licensing assessment, standardized ship models should be used. The current lack of standardized mathematical models makes it difficult to evaluate mariner performance in ship maneuverability. Simulations used for licensing assessment and qualification of shiphandlers would benefit from being highly accurate with respect to trajectory prediction and overall realism.

A standard set of harbor operating conditions could be developed for specific ships that can be accurately validated against measurements of performance and used as a consistent basis for assessing mariner competency. Since accurate ship and channel-environmental models are required for some purposes, it would seem reasonable that such models be used for all purposes. Current mathematical models could be modular and validated in parts, so that accuracies could be established.

Once designed, the exercise scenarios need to be validated through cross-platform and cross-student research. For cross-platform validation to be possible, there need to be industrywide standards for the different levels and uses of the simulations. If a simulation is used for licensing assessment, it is crucial that any performance variations are recognized and accounted for during assessment.

If manned models are used for licensing assessment, validation will be needed to ensure that models produce faithful vessel behavior.

Objective and Subjective Validation

The approach to validation of marine simulators or simulations might follow the commercial air carrier industry's approach of including objective with subjective evaluation. Model validation requirements would be based on simulator-based training and licensing objectives.

Objective Validation

The elements of a simulation that require validation are the accuracy and fidelity of:

  • image portrayal, including the content, quality, field and depth of view, and movement of the visual scene;
  • the predicted ship trajectories based on hydrodynamic and aerodynamic modeling;
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
  • own ship (ship model) characteristics; and
  • the operational scenarios used for evaluating or assessing.

These quantifiable factors can be measured and incorporated into an overall "realism rating" that can be compared among simulator facilities.

Ship maneuvering can be validated through a formal, objective process. Standardized models are selected and tested in towing tanks and the results compared to selected full-scale real-ship trials of the same ships to provide bench-mark data for validation and testing of simulators.

Subjective Validation

Subjective validation of the simulator and simulation should extend beyond the "test mariner" program to involve the use of panels of credible experts to assess whether or to what degree a simulation consistently results in behavior that would be expected under identical or similar conditions in the real-world. Such panels are especially important for any type of formal training evaluation or licensing assessment. Impartial validation panels need to be carefully composed of people from multiple disciplines, including, for example:

  • a simulation instructor or assessor,
  • a representative of the licensing authority,
  • a subject-matter expert, and
  • an expert to validate ship-model behavior.

An Approach to Validation of Ad Hoc Models and Modifications

There are strong reasons to discourage ad hoc creation or field adjustments of mathematical models using software within some simulators because field adjustments almost invariably introduce uncertainties and unanticipated inaccuracies.

The commercial air carrier industry requires that all modifications to a simulation be documented and that the simulator be re-evaluated after modifications are made. Within the marine industry, it may be possible to establish a simulation baseline through initial validation, then require documentation and revalidation as modifications are made. The program might include requirements such as:

  • Creations or modifications of models must be undertaken by an interdisciplinary team of individuals qualified in both the operational and hydrodynamic elements of the simulation, or
  • Field adjustments or changes in model behavior should be reviewed by a suitably qualified hydrodynamicist and an external mariner to validate hydrodynamic fidelity.

Procedures are needed to ensure that such changes are documented and that the original vendors of the data and appropriate authorities are notified.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

Process for Developing Industrywide Standards

Standards can be issued by any organization whose authority is accepted in the subject area, either government or nongovernment, but they require a great deal of cooperation on all levels. "A regulation is a relatively short step from a standard, but an important one. Many times standards are converted to regulations with little or no change in text" (NRC, 1985). It is, therefore, very important that standards for simulators and simulations be carefully developed, including consideration of the views of all parties of interest.

There are a number of organizations actively interested in the development of simulator standards. Professional groups, such as the International Marine Simulator Forum and the International Maritime Lecturers Association, have been developing technical standards for simulators. The current revision of the international Standards for Training, Certification, and Watchkeeping guidelines is expected to contain guidelines for simulators, but is not expected to propose technical specifications in the near future (Drown and Mercer, 1995).

Within the United States, the process for developing standards is well developed. The American National Standards Institute (ANSI) is an organization whose principal function is to recognize and respond to needs for standards and to arrange for involvement by its standards-development members. ANSI can play a very effective role in serving as the focal point for communications between its members who develop standards and those in the private sector and government who need and use standards (NRC, 1985).

Other professional societies, such as the Society of Naval Architects and Marine Engineers and the American Society for Testing and Materials, might also have an interest in assisting in the development of industrywide standards. For simulator and simulation standards, there are numerous parties at interest, including facility operators, shipping companies, unions pilot associations, port authorities, regulators, maritime academies, training schools, and the mariners themselves. To be effectively implemented, the simulator and simulation standards must be based on a consensus view.

It may also be of value for simulator manufacturers and facility operators to subject their activities to International Organization for Standardization 9000 quality assurance certification. Such certification would help ensure that minimum standards are being maintained by these organizations.

ISSUES AND FUTURE DEVELOPMENTS

Validity Issues

Because demonstrations might have high face validity but lack the rigor and thoroughness needed to determine the accuracy of the trajectory prediction capability, demonstrating the simulator or simulation is not a substitute for formal

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

validation. Ultimately, in the use of simulation for training performance evaluation and licensing assessment, the employer or regulatory authority needs to consider which of the following factors bear on the simulation's validity:

  • whether the trajectory prediction models were provided by a modeler or developed independently by the simulator facility;
  • the manner and process by which the performance of the model or models were validated;
  • the identity and credentials of the validators;
  • whether the trajectory prediction model or models were adjusted since they were last validated;
  • the accuracy that can be achieved with the accuracy prediction model;
  • the quality of the bathymetry and hydrographic data that drive the waterway model used in the simulation (of special importance if the trainee will use the results of port-and waterway-specific training in that port or waterway) and how these data were obtained and validated; and
  • the validity of the model's behavior.

Future Simulation Development

Vessel trajectory prediction modeling is a developed science that provides highly useful tools for building marine training simulators. From a technology perspective, the future is promising for improving the accuracy, flexibility, and extent of simulator-based training and licensing assessment applications. Theoretical and numerical methods are powerful and nearing practical application. Computational power, graphics, and multimedia capabilities and the proliferation of microcomputers enable the use of sophisticated mathematical trajectory models.

FINDINGS

Summary of Findings

The accuracy and fidelity of ship-bridge simulators can vary significantly from facility to facility. These differences derive from differences among original mathematical models used to develop the simulations and facility operator modifications to models after installation.

There are no industrywide simulator or simulation standards. Currently, simulations are initially validated by the manufacturer, then validated by the facility operator through use of subjective "test mariners." Often, facility operators continue to periodically modify simulations after the initial validation. This practice of continually modifying simulations can result in inconsistent training programs, as successive classes may be conducted with different simulations. These problems

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

are of particular concern when a simulation is used for licensing or training related to remission of sea-time.

To address these concerns, simulators and simulations should be validated based on proposed use and required level of performance. All modifications should be documented and the simulation revalidated. The extent to which accuracy of a simulation needs to be validated should depend on its proposed use.

An approach to simulator and simulation validation might follow the commercial air carrier industry process, which relies on both objective and subjective validation. The objective validations could include development of a "realism rating" based on an assessment of factors such as image portrayal and predicted ship trajectories. The subjective validations could be conducted by an impartial panel of experts.

Many organizations and individuals have an interest in the development of these standards. There are a number of national and international organizations that could effectively work toward the consensus necessary to promulgate standards that would be acceptable to all parties.

Research Needs

There are number of areas where additional research in hydrodynamic and mathematical modeling could be applied to improve the accuracy of simulators and simulations. On a process level, value could be gained from the development of standard method for the exchange of models or modeling coefficients. Also, if models were developed in modules, validation of the simulations and simulation characteristics modifications would be facilitated, as would replacement of outdated modules. Finally, simulation models could be improved through greater public access to existing proprietary hydrodynamic and other towing-tank data.

Additional research is needed to extend databases in areas such as ship maneuvering coefficients and anchoring evolutions. Also, the application of computational methods for determining pertinent hydrodynamic parameters offers great promise as a solution to current problems in modeling and ship-bridge simulators. Ship modeling would be improved through development and implementation of research codes to be applied to specific computational tasks of maneuvering simulation.

Full-scale real-ship experiments would advance the state of practice in modeling, particularly for shallow and restricted waters with banks. Modeling for operations at slow speed and with reversing propeller situations also needs improvement.

REFERENCES

Drown, D.F., and R.M. Mercer. 1995. Applying marine simulation to improve mariner professional development. Pp. 597–608 in Proceedings of Ports '95, March 13–25, 1995. New York: American Society of Civil Engineers.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×

Hays, R.T., and M.J. Singer. 1989. Simulation Fidelity in Training System Design: Bridging the Gap Between Reality and Training. New York: Springer-Verlag.


NRC (National Research Council). 1985. Status of marine and maritime standards. Steering Committee on Engineering Standards for Marine Applications, Marine Board, Commission on Engineering and Technical Systems, Washington, D.C.

NRC (National Research Council). 1992. Shiphandling Simulation: Application to Waterway Design. W. Webster, ed., Committee on Shiphandling Simulation, Marine Board. Washington, D.C.: National Academy Press.

Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 158
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 159
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 160
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 161
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 162
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 163
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 164
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 165
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 166
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 167
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 168
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 169
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 170
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 171
Suggested Citation:"7 Simulation and Simulator Validity and Validation." National Research Council. 1996. Simulated Voyages: Using Simulation Technology to Train and License Mariners. Washington, DC: The National Academies Press. doi: 10.17226/5065.
×
Page 172
Next: 8 Conclusions and Recommendations »
Simulated Voyages: Using Simulation Technology to Train and License Mariners Get This Book
×
Buy Paperback | $52.00 Buy Ebook | $41.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

This book assesses the state of practice and use of ship-bridge simulators in the professional development and licensing of deck officers and marine pilots. It focuses on full-mission computer-based simulators and manned models. It analyzes their use in instruction, evaluation and licensing and gives information and practical guidance on the establishment of training and licensing program standards, and on simulator and simulation validation.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!