8
Feedback to Office of Emergency Preparedness on Program Success

Regardless of how the Metropolitan Medical Response System (MMRS) program is managed by the U.S. Department of Health and Human Services’ Office of Emergency Preparedness (OEP), the ultimate test of the program’s worth lies in how well it has helped local communities prepare for the consequences of a massive terrorist attack with chemical, biological, or radiological (CBR) weapons. The survey described in the previous chapter begins to answer this question by soliciting the opinions of the communities themselves. This chapter complements that approach by presenting the recommendations of the Committee on Evaluation of the Metropolitan Medical Response System Program for an independent and systematic assessment of the response capabilities of the large metropolitan areas that have or will participate in the MMRS program. The title to the chapter was chosen to emphasize an important assumption or guiding principle of the committee: that program assessment is primarily for the purpose of identifying and correcting shortfalls in OEP’s MMRS program. Several other interrelated principles also underlie the chapter and the committee’s recommendations:

  • Evaluation should be part of a continuous learning and continuous quality improvement program, not a one-time snapshot. This implies a continuing relationship between the communities and their evaluators that includes financial as well as technical and educational support.

  • “Preparedness” is a meaningless abstract concept, since threats vary among communities and change over time, perhaps even in response



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program 8 Feedback to Office of Emergency Preparedness on Program Success Regardless of how the Metropolitan Medical Response System (MMRS) program is managed by the U.S. Department of Health and Human Services’ Office of Emergency Preparedness (OEP), the ultimate test of the program’s worth lies in how well it has helped local communities prepare for the consequences of a massive terrorist attack with chemical, biological, or radiological (CBR) weapons. The survey described in the previous chapter begins to answer this question by soliciting the opinions of the communities themselves. This chapter complements that approach by presenting the recommendations of the Committee on Evaluation of the Metropolitan Medical Response System Program for an independent and systematic assessment of the response capabilities of the large metropolitan areas that have or will participate in the MMRS program. The title to the chapter was chosen to emphasize an important assumption or guiding principle of the committee: that program assessment is primarily for the purpose of identifying and correcting shortfalls in OEP’s MMRS program. Several other interrelated principles also underlie the chapter and the committee’s recommendations: Evaluation should be part of a continuous learning and continuous quality improvement program, not a one-time snapshot. This implies a continuing relationship between the communities and their evaluators that includes financial as well as technical and educational support. “Preparedness” is a meaningless abstract concept, since threats vary among communities and change over time, perhaps even in response

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program to a community’s level of preparedness; readiness should be seen as a process rather than a state. Preparedness requires not only numerous specific capabilities, typically the responsibilities of independent offices, agencies, and institutions, but also seamless coordination of those capabilities into a coherent response. The former may be envisioned as the teeth of a comb, the latter as the base or backbone of the comb. Information and the ability to acquire, process, and appropriately distribute it to essential sites and personnel are central to the effective management of critical incidents including terrorism in its many forms. Evaluation is an exercise designed to guide the distribution of local, state, and federal resources. Evaluations should be valued and understood as an opportunity for local communities to determine the areas in need of improvement and support rather than as a test of communities’ self-reliance. Evaluation by OEP should be a multilevel process that includes (1) periodic review of documents and records, (2) observation of community-initiated exercises and drills, and (3) on-site assessment. The committee views the on-site assessment as comprising both interviews of individuals about specific capabilities and a scenario-driven group interaction focused on cooperation and coordination. A relatively small subset of the nearly 500 preparedness indicators identified in the Phase I report (Institute of Medicine, 2001) can be used to identify critical areas in need of improvement for a given community. As noted earlier in the report, in the absence of any proper control cities or pre-MMRS data, it will be impossible to unequivocally assign credit to OEP for high states of preparedness. Most of the larger cities have received training and equipment from the U.S. Department of Defense or the U.S. Department of Justice, some have received grants and training from the Centers for Disease Control and Prevention (CDC), and all have spent time and money from state and local budgets. The MMRS program’s emphasis on multiagency, multijurisdictional planning undoubtedly played a major role in increasing preparedness in many cities, but no large city could become well prepared solely as a result of the relatively meager funding provided by the OEP contracts. The remainder of this chapter describes a three-element evaluation procedure built upon these principles. The three elements are review of written documents and data, a site visit by a team of peer reviewers, and observations at exercises and drills. The three procedures are complementary means of analyzing the community’s response capabilities, and the next two sections focus on first identifying a subset of essential capabilities and then specifying preparedness criteria for each. The chapter con-

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program cludes by proposing some specific procedures for gathering data at exercises and drills and at site visits. ESSENTIAL RESPONSE CAPABILITIES As noted above, the committee’s Phase I report (Institute of Medicine, 2001) identified nearly 500 potential indicators of preparedness (see Appendix E). Chapter 6 provides information on how and why the committee arrived at these indicators, suggests that no evaluation would be likely to use all of the defined indicators, and proposes a means of beginning the selection problem, namely, to look first for output measures and to rely on process and input measures only when corresponding outputs are unavailable. This approach greatly reduces the number of potential items to be included in an evaluation effort, but the evaluator is still left with a large number of items. The committee has made further reductions by analyzing the critical actions required for effective responses to large-scale CBR terrorism incidents, that is, the essential response capabilities. The specific characteristics and importance of these essential response capabilities vary with the type of agent and the other details of the incident, as do the relative importance of the various capabilities, but the many elements on the list of MMRS program contract deliverables and the corresponding preparedness indicators can be integrated into a list of 23 essential functions. They are listed below in the order in which they would generally become necessary: Relationship development (Communication, coordination, and control are especially critical in responding to events like those of the autumn of 2001) city agencies, state and other local governments federal agencies and local federal facilities private institutions, especially health care institutions voluntary community organizations (e.g., Red Cross, churches, ham radio operators) Communication system development telephonic, computer, and radio hardware; points of contact; procedures alternatives to commercial services, which are likely to be overwhelmed during a terrorist event. mutual aid pacts with nearby communities address compatible communication equipment

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Hazard assessment high-risk sites identified and contingency plans developed (although OEP asks for plans for coping with 1000 victims of a chemical attack and 3 levels of biological attack, each community should use this assessment to make estimates of likely casualty volumes specific to the area and situation). Training awareness, equipment, treatment, exercises fire and police departments, emergency medical technicians, emergency services and public health personnel, hospitals, individual medical care providers Equipment and supplies purchase and maintenance (general purpose as well as specialized for CBR agents) reception and distribution of “push package” from CDC Mass immunization and prophylaxis Addressing the information needs of the public and the news media (Experience from September 11 and the anthrax incidents emphasizes the importance of early and frequent communication with the public through a single authoritative spokesperson on health matters.) First responder protection Rescue and stabilization of victims Diagnosis and agent identification hardware and software to monitor health trends in close to real time Decontamination of victims (at site of exposure or at a hospital or treatment site) Transportation of victims from incident site to hospital or casualty collection point from hospital to hospital (including use of National Disaster Medical System [NDMS]) patient tracking system

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Distribution of supplies, equipment, and pharmaceuticals from local cache from National Pharmaceutical Stockpile Shelter and feeding of evacuated and displaced persons provisions for emergency shelter for persons fleeing sites of perceived danger Definitive medical care trained personnel, beds, supplies and equipment locations, in event that existing hospitals’ capacities are inadequate mass immunization or distribution of drugs or vaccines Mental health services for responders, victims, caregivers, and their families Volunteer utilization and control Crowd and traffic control at and near facilities rendering emergency medical care at hospitals or facilities dispensing medication along evacuation routes Evacuation and quarantine decisions and operations Fatality management large numbers of contaminated or infectious corpses Environmental cleanup, physical restoration of facilities, and certification of safety Follow-up study of responder, caregiver, and victim health Process for continuous evaluation of needs and resources, during and after exercises and actual CBR, mass-casualty, and hazardous material (hazmat) events and disease outbreaks With these 23 essential functions or capabilities as a guide, selection of a set of preparedness indicators for evaluation is considerably easier. The committee believes that the set of indicators described in the following section can serve as a suitable proxy for preparedness. Evaluating even this limited set of indicators nevertheless demands evaluators or auditors

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program with diverse backgrounds and expertise in a variety of areas as well as several auditing techniques. The approach taken by the committee calls for a combination of evaluation of documents submitted to OEP by the community to be evaluated, on-site questioning by a site-visit team, and direct observation of drills and exercises. PREPAREDNESS INDICATORS FOR EVALUATION OF WRITTEN SUBMISSIONS, ON-SITE INSPECTION, AND OBSERVED EXERCISES Table 8-1 shows a list of preparedness indicators selected from the tables of Appendix E that cover the 23 essential MMRS capabilities outlined in the previous section. In accordance with the philosophy expressed at the beginning of this chapter, the committee sought to keep the number of indicators requiring on-site interviews or observations to a manageable number by selecting a number of indicators that involved documents or other written records that could be mailed or otherwise sent to the evaluators. For many essential capabilities, such written records were the optimal or only feasible indicators; in other cases, selection of a written evaluation rather than an on-site evaluation was a compromise between the optimal measure and the realities of time and expense for both the assessor and the assessed. As suggested in Chapter 6, the committee considers output indicators more likely to be valid than process and input indicators, and Table 8-1 therefore heavily favors output indicators for on-site evaluations. Should any of the listed output indicators be unavailable, the committee expects the evaluator to use the corresponding process or input indicators to guide a judgment about preparedness. PREPAREDNESS CRITERIA No generally agreed-upon model for local preparedness exists, nor are good data or even a solid consensus available about what constitutes acceptable objective evidence of capability for most of the preparedness indicators listed above. The committee is also sensitive to the great diversity of circumstances facing the nation’s cities and the variety of ways in which they are organized to respond to emergencies. The committee therefore chose not to try to specify rigid standards for each of the countless possible combinations of incident types and response approaches. Instead, the proposed assessment program puts considerable faith in the judgment of what is now a relatively small but rapidly growing cadre of individuals who have been in the forefront of responding to and planning for responses to incidents related to the use of CBR agents. Nevertheless, to ensure the comparabilities of the assessments made by different evalu-

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program ators and to provide communities with some indication of what those experts will be looking for, the committee has assembled below, for each of the essential capabilities listed in Table 8-1, criteria that should serve as the core of evaluators’ judgments about preparedness. Armed with these criteria, it should be possible for an evaluator or a team of evaluators to determine whether a community is well prepared, prepared, or poorly prepared in each of the 23 essential capabilities. The committee feels strongly that the real value in the proposed evaluation system lies in identifying capabilities in need of improvement, and has therefore resisted the temptation to specify a passing grade. The fact is that, as noted on page113, preparedness as an abstract concept is meaningless. That is, even a perfect score on each of the 23 capabilities will not guarantee an optimal response to all imaginable CBR events; less than a perfect score makes it easy to imagine an event that will be handled poorly. 1. Relationship Development (Partnering) Preparedness Indicator Documentation of effective coordination in an exercise or an actual incident with or without CBR agents. The lead agency provides written documentation of an MMRS-wide response system that includes management, operations, logistics, planning and intelligence, and finance and administration activities. The lead agency provides a list of community-level response plans used regularly for non-terrorist-related emergencies or disasters, for example: Emergency Operations Plan Multiple Casualty Incident Plan Hazmat Response Plan Emergency Medical Service (EMS) Management Plan The lead agency provides an after-action report from a full-scale exercise, which should be conducted at least once every 3 years. The lead agency documents actual events, including evaluation of potential for terrorism by the incident commander. Preparedness Indicator Evidence from exercises or actual events demonstrating workable interface among local plans.

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program TABLE 8-1 Preparedness Indicators and Mode of Evaluation of MMRS Plan Elements Relevant to Each of 23 Essential Capabilities Essential Capability MMRS Plan Element 1. Relationship development (partnering) 2.02 Description of how responses to a CBR terrorism incident by public safety, public health, and health services sectors will be coordinated   3.04 Coordination with other political, mutual-aid, or other MMRS program jurisdictions 2 Communication system development 3.07 Detailed notification and alert procedures via redundant systems           3.09 Provisions for accurate and timely dissemination of information among MMRS members   8.01 Procedures for notification of hospitals, clinics, health maintenance organizations (HMOs), etc., that an incident has occurred       8.New 1 Procedures for recall of staff     3. Hazard assessment 2.New 1 Description of the planning environment (i.e., identification of local hazards, baseline strengths, vulnerabilities) 4. Training 3.20 A schedule for exercises   9.01 Training requirements for all personnel responding to the scene of an incident or providing care to victims of a CBR agent-related incident    

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Preparedness Indicator Written Site Visit Drill Documentation of effective coordination in an exercise or an actual incident with or without CBR agents X   X Evidence from exercises or actual events demonstrating workable interface among local plans X X X • Documented success by regular testing or during actual use in an emergency X   X • System is not dependent on commercial services alone. X X   • Mutual aid agreements with surrounding communities insure interoperability of key communication systems X     Demonstration of effective use of all systems in periods of peak demand through unannounced tests or use in an actual emergency X   X • Percentage of facilities contacted in 1 hour during weekly notification checks X     • Time from initial contact to initiation of hospital disaster plan or incident command system X   X • Calls to random sample of list demonstrate that list is up to date   X   • Percentage of staff returning calls in 2 hours X   X A communitywide assessment identifies strengths, barriers and challenges, and a priority list for planning efforts X     Collection of after-action reports X     • Demonstration of knowledge of subject matter to peer reviewer by selected sample of trained personnel from any level of any participating organizations or through functional drills, communitywide exercises, or responses to actual CBR agent, hazmat, or infectious disease outbreak events   X X • Certification or other nationally recognized affirmation of CBR agent-specific knowledge and skills, if such means for certification become available in the future X    

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Essential Capability MMRS Plan Element 5. Equipment and supplies 10.02 Quantities of pharmaceuticals sufficient to care for 1,000 victims of a chemical agent and for entire affected population for 24 hours after a biological incident 10.04 Detailed procedures for equipment maintenance and pharmaceutical storage                 6 Mass immunization or prophylaxis 7.05 New plans or augmentation of existing plans for management and implementation of a mass immunization or prophylaxis plan         7. Attention to the information needs of the public and the news media 3.08 Detailed management procedures for public affairs         8. First responder protection 5.11 Procedures for procurement and provision of appropriate equipment and supplies 8.05 Availability of adequate personal protective equipment for hospital and clinic providers

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Preparedness Indicator Written Site Visit Drill Availability of all required antidotes, antibiotics, and immune sera, in appropriate quantities, for inspection by site-visit team or peer reviewer   X   • Evidence that the mechanism of delivery and storage is secure in natural disasters, mock drills, earthquakes, or hazmat events   X   • Consistency of inventory with records of pharmacy and therapeutics committee meetings X     • Knowledge of procedures for return of unused supplies and decontamination of equipment by logistics personnel   X   • Evidence that a sample of equipment selected by peer reviewer is in working order   X   • Performance of required maintenance and/or prompt retrieval of maintenance manual by logistics personnel when queried by peer reviewer   X   • After-action report detailing successful response to a CBR incident (real or a hoax), a natural outbreak of disease (e.g., a meningitis or influenza vaccination campaign or an outbreak of rabies or giardiasis), or a large-scale exercise X     • Identification in the plan of distribution sites and required personnel X     • Percentage of responder and caregiving personnel immunized if the plan calls for prophylactic immunizations X     • A knowledgeable and credible spokesperson has been designated to provide health information to the public in the event of a terrorist attack with a CBR agent. X     • Collection of finished communiqués X     • Documented use of media packages in CBR agent-related hoaxes or incidents or other hazmat-related or epidemic events X     Demonstration that the appropriate types and quantities of equipment and supplies have been purchased and are readily accessible   X X Demonstration of competency with equipment (e.g., by a respirator fit test) for expert peer reviewer   X X

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program Preparedness Indicator Officers receive regular training in crowd control. Every officer should receive basic training in the police academy and then refresher training every 2 or 3 years. The training should cover, among other things: expectations for various types of events, crowd control measures, understanding of the impact of deploying various chemical munitions, and the use and testing of gas masks. Other officers should receive more specialized training at a much greater frequency. Many police agencies use “field force” techniques developed in Miami and Dade County, Florida, in the 1980s following several riots. These officers should receive initial training of 2 or 3 days and then refresher training at least once a year (in Charlotte, North Carolina, about 400 officers have received this training). Special Weapons and Tactics (SWAT) teams should train every month to deal with armed and barricaded subjects and high-risk entries. Any specialized team that would be called out during a situation involving a CBR agent should receive training and conduct exercises on a monthly basis. Lesson plans, supporting procedural documents, attendance records documenting participation and proficiency, schedules for future training, and after-action reviews of exercises should all be available. Preparedness Indicator Adequate protective equipment is available for police officers. Specialized teams that would be called out during a situation involving a CBR agent and that include police should be equipped with personal protective equipment ranging from level D to level A. Depending on the community, one might see all officers with gas masks and helmets for use in riot control situations. Properly fitted gas masks might provide a minimal level of short-term protection against some hazards. Field force officers should be fully equipped with crowd control gear, in addition to gas masks and helmets. The “appropriate” level of protective gear for police officers not in a designated response role is difficult to establish. At a minimum, an

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program assessment has been made and action taken to provide whatever personal protective equipment that assessment supports. 19. Evacuation and Quarantine Decisions and Operations Preparedness Indicator A written plan that includes procedures for deciding upon and implementing public safety measures such as providing shelter in place, conducting an orderly evacuation, quarantining of individuals and geographical areas, and isolating patients or groups of patients. The plan does not need to be specific to CBR agent-related events. Legal authority for the decision maker is established and documented in the plan. Plans provide for the medical care of quarantined individuals. Plans provide for the nonmedical care of quarantined or isolated individuals. A written plan for the media includes prepared information and fact sheets explaining the need for and processes for the implementation of evacuation or quarantine. Preparedness Indicator Identified leadership can verbalize the contents of the procedure for the evacuation of a contaminated facility. The responsible entity has reviewed the experience of the state, county, and city with the evacuation of facilities that have occurred during previous emergency or urgent conditions. The evacuation plan has been practiced during disaster exercises (as evaluated by examination of after-action reports). Preparedness Indicator The identified leadership can verbalize the contents of the procedure for the initiation of isolation or quarantine. The responsible entity has reviewed the experience of the state, county, and city with isolations or quarantines that have occurred during previous public health operations. The isolation or quarantine plan has been practiced during disaster exercises (as evaluated by examination of after-action reports).

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program 20. Fatality Management Preparedness Indicator Contingency contracts or other arrangements for storage capacity with local hospital morgues, mortuaries, warehouses, other facilities with cold-storage capabilities, and sources of refrigerated trucks. Preparedness Indicator Evidence of NDMS support and procedures for activation. The lead agency provides evidence of joint training or tabletop demonstration of interface with Disaster Mortuary Operational Response Teams. Preparedness Indicator Hands-on demonstration of decontamination can be provided in an exercise or actual incident. The MMRS plan identifies required decontamination equipment and the basis for the choices of particular types and quantities of equipment. The equipment chosen is adequate to support the decontamination of up to 1,000 victims of a terrorist incident involving chemical agents. Inspection of at least one site confirms that the equipment is in inventory and is readily accessible. The on-site inventory complies with the plan. If the equipment is stored in a locked area, staff can locate the key without assistance. The necessary equipment can be set up and functioning within 30 minutes of arrival on site. Procedures for expedient decontamination and keeping ambulatory victims on site for 30 minutes are in place. Equipment setup is not dependent solely on members of the hazmat unit. Training required for both setup and operation has been provided to enough personnel to ensure the capability of equipment operation at all times (24 hours a day, 7 days a week, every day of the year). Preparedness Indicator Evidence that standard operating procedures are available at morgue fa-

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program cilities in sufficient quantity to distribute to any expedient sites and that the required personnel are available. 21. Environmental Cleanup, Physical Restoration of Facilities, and Certification of Safety Preparedness Indicator Demonstration of an effective process for the identification of environmental risk and determination of the need for decontamination or vector intervention in response to questioning by the peer reviewer or by performance in an exercise, actual hazmat event, or disease outbreak. Review of the MMRS plan shows that it includes provisions for determination of risk and the need for decontamination or vector intervention and patient treatment. The agencies and organizations required to do the following tasks have been identified: determine the existence and nature of hazardous materials or the existence and nature of vectors, communicate findings to all MMRS response and management elements, communicate messages to the public, and iv. carry out long-term surveillance and cleanup of the affected area, as required. Agreements are in place to secure additional (decontamination) response elements (personnel, supplies, and equipment). A training and exercise program is available to support the system and protocols. An on-site visit to a hazmat response team, an EMS unit, a hospital ED, or some other organization is made to observe the procedures and protocols used to identify environmental risk and determine the need for decontamination or vector intervention. A sample of personnel is able to demonstrate the use of detection and agent identification equipment, demonstrate the use of personal protective equipment, and demonstrate use of a field management system for incorporation of specialty environmental resource agencies into the MMRS plan.

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program 22. Follow-up Study of Responder, Caregiver, and Victim Health Preparedness Indicator The response plan includes a practical process for scientific investigation of human health effects in responders, caregivers, and victims. The local public health system has a written protocol for follow-up investigation of the human health effects (short term and long term), including the following: an assessment of the nature and magnitude of the incident, including the agent(s) involved and the population affected; a process for assessing the availability of resources, including the appropriate personnel, equipment, budget, treatment, and laboratory capacities; baseline health data on designated emergency response personnel; the identification of the “study” population and a control population, if appropriate; and a communications strategy for reporting the process and results of the study to the community. 23. Process for Continuous Evaluation of Needs and Resources Preparedness Indicator Possession of a collection of after-action reports by all participating agencies and institutions. On-site indications that key agency participants have actually received after-action reports are available. After-action reports for exercises and major events requiring emergency management are available. They should include, at a minimum: a description of the exercise or incident, the objectives of the exercise, the roles played by various agencies and key individuals (public and private, both inside and outside the governmental unit preparing the report), a list of problems or shortcomings encountered, an assessment of the reasons that these problems or shortcomings occurred, and an analysis of lessons for improved future performance.

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program In the absence of relevant after-action reports, written indications should assign responsibility for preparation of the reports to a particular agency or individual or should describe a procedure for assigning ad hoc responsibility in advance of planned exercises or immediately after the event for unplanned emergency management situations. Preparedness Indicator During-action reports from extended exercises or prolonged responses to actual CBR agent or hazmat events On-site indications that key agency participants have actually received reports are available. Reports should include, at a minimum: a description of the exercise or incident, the objectives of the exercise, the roles played by various agencies and key individuals (public and private, both inside and outside the governmental unit preparing the report), a list of problems or shortcomings encountered, an assessment of the reasons that these problems or shortcomings occurred, and an analysis of alterations in the community response required for meeting evolving needs. In the absence of relevant during-action reports, written procedures should assign responsibility for preparation of the reports to a particular agency or individual or should describe a procedure for assigning ad hoc responsibility for monitoring planned exercises or unplanned emergency management situations for unanticipated developments. Preparedness Indicator Evidence for changes in structure or functioning in response to reported deficiencies. A distribution list or lists for different types of after-action reports should be available so that findings can be disseminated to participants, supervisors, and policy officials. A procedure for securing reviews of and comments on reports by other participants or close observers of the events covered should be in place. At a minimum, these should be in writing. For major exercises or events, provision should also be made for in-person discussions by key agency officials.

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program The lead agency provides files of written comments and minutes of meetings that discuss the findings, including evidence of agreement on steps taken in light of identified problems or shortcomings. A procedure for generating and assigning responsibility for recommended steps to maintain or improve preparedness should be in place. Procedures for assigning responsibility for subsequent follow-up should be in place to see whether the proposed steps have been taken or, if not, whether the problems or shortcomings have been addressed in another appropriate way. Memos or meeting minutes indicating the following should also be available: that subsequent follow-up of the steps has occurred and that any incomplete steps are still being monitored, that substitute actions have been scheduled, or that analysis of obstacles and a search for workable solutions are ongoing. EXERCISES AND DRILLS The committee members began their task with the common view that, in the absence of regularly occurring CBR terrorism incidents, the plans produced by MMRS program cities might be best evaluated by large-scale field exercises that would simulate such incidents and more specialized drills that would test the performances of specialized portions of the overall response plan. A proposal debated early in the committee’s discussions was to design an exercise(s) that would constitute a comprehensive test of each city’s response plan. The evaluation would then simply involve conducting the exercise and observing the response. This proposal was ultimately rejected as being too expensive in terms of the financial cost for OEP and in terms of time for local emergency response and medical personnel, difficult to tailor to 100 different locales, and in the case of a covert release of a biological agent, impossible to simulate realistically and ethically. Several members also observed that in their experiences it had been the planning rather than the conduct of exercises that was of greater value to the community. One of the MMRS program contract deliverables in fact calls for a schedule of exercises, and another calls for the collection and distribution of after-action reports, so the committee opted to incorporate these exercises into the overall evaluation plan. Observers, preferably members of the team that will subsequently conduct a site visit to a community conducting an exercise, should attend large-scale exercises and significant drills before they plan a site visit. Despite the drawbacks mentioned in the previous paragraph, many of the essential capabilities can best be assessed

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program in this fashion, and some can only be assessed in this manner. Table 8-1 and the associated preparedness criteria can serve as guides for these observers, who should be required to produce written reports of their observations and judgments of preparedness for each of the essential capabilities with an X in the column of Table 8-1 labeled “Drill.” Given the expense and difficulty of planning and conducting a large scale exercise, OEP should also consider sharing these observations, suitably redacted to maintain security, with other MMRS program cities, perhaps by means of its password-limited website SITE VISITS AND PEER EVALUATORS Although the details of any site visit to some extent will be specific to the site being visited, the committee envisions a typical site visit consisting of a 3-day evaluation. The assessment team would gather on the afternoon or evening of Day 1 to meet, confirm assignments, and distribute the required materials. Day 2 would be devoted to individual interviews and observations, as would the morning of Day 3. Two scenario-driven group discussions would take place simultaneously on the afternoon of Day 3 (see below for more detail), and at least two assessment team members would attend each scenario. Debriefing of the team (i.e., when team members discuss their observations with each other) would take place on the morning of Day 4, and on the afternoon of Day 4, the team would debrief the community (i.e., provide some very general feedback on the team’s observations and conclusions). A formal report would be produced in the ensuing month by OEP staff or their representatives and would be based on the collective observations of the assessment team. The assessment team should consist of five individuals collectively experienced in a variety of disciplines and professions. Their task is a broad one, and it is important that they be, and be perceived as, peers of the individuals being assessed. To this end the committee recommends that the team comprise a fire department representative familiar with hazmat operations; a city- or county-level emergency manager; a local public health officer familiar with surveillance systems; an individual with extensive managerial, operational, and clinical experience in the field of prehospital emergency medical services; and an acute-care medical practitioner, who could be a nurse or a physician, with clinical experience in infectious diseases or emergency medicine and mass-casualty operations. In practice, such a team would no doubt need one or two administrative support personnel. Consideration should also be given to including OEP’s regional Public Health Service emergency coordinator on the assessment team. This individual generally has served as the contracting officer’s technical representative for the MMRS program contract. Inclusion on the

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program team may produce a conflict of interest for the emergency coordinator, but he or she will also bring substantial important information regarding the local MMRS program. Similarly, the committee recommends that at least three of the five members have some previous involvement with the MMRS in their own community. Both the community and the prospective site visitors should be notified at least six months in advance of the anticipated visit. This will allow both the community and the site visitors time to make necessary logistical arrangements, gather documents, and arrange schedules of likely participants. It will also allow OEP to gather necessary documents, review reports from previously observed exercise and drills in the community, and schedule some pre-visit training for the site visitors, which will be vital to insuring a consistent and valid assessment program. Some pilot testing will be necessary to confirm the feasibility of this suggested approach, check interobserver reliability, and make changes where the committee’s suggestions cannot be implemented. In doing this pilot testing OEP should endeavor to include communities it has some reason to believe lie at of the extremes of the preparedness continuum. It seems unlikely that all MMRS communities are equally well prepared, despite OEP intentions, and an assessment program should at least be able to distinguish the extremes of systemic or societal preparedness. Finally, it should be obvious that the assessment program being proposed here will entail considerable expense (comparable site visits to Urban Search and Rescue Teams cost approximately $30,000 each, and OEP has already let more than 100 MMRS contracts). The program will also make substantial demands of the time of OEP staff; the committee believes this task will necessitate at least one professional position. SCENARIO-DRIVEN GROUP INTERACTION Every site visit will involve not only individual interviews and observations but also two simultaneous 3-hour group meetings, each facilitated by two on-site evaluators, in which a group of 12 to 15 representatives from the community’s safety and health institutions will be required to answer questions about their community’s response to a fictional CBR terrorism incident. The models and scenarios are adaptations of three FEMA courses designed to help senior local government officials improve their abilities to respond to mass-casualty incidents involving the use of CBR weapons (Federal Emergency Management Agency, 2001c, d, e, f). Because of the overarching importance of interagency, intergovernmental, and public-private cooperation and coordination, the goal of this portion of the site visit is to give the community a chance to demonstrate the

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program existence of a well-understood process to coordinate all necessary capabilities to respond to a mass-casualty CBR terrorism incident, specifically, the ability to acquire, process, and appropriately distribute information required to effectively manage critical functions during an incident. The fact that the evaluators will conduct two parallel discussions will ensure that this ability is not confined to a single individual or a single individual in each institution. Appendix F provides scenarios, discussion questions, and instructional material for the facilitators and evaluators (these roles should be assigned to different members of the site visit team). The scenarios involving chemical and radiological agents have been taken from the FEMA courses almost intact, but the FEMA scenario involving a biological agent, which involves an attack with anthrax, has been extensively modified to reflect both the knowledge about anthrax gained in the autumn of 2001 and the committee’s desire to include a scenario based on a truly covert release of a biological agent. The materials from the FEMA courses are designed to support either 3-day or 1-day courses, so considerable editing of discussion questions was necessary. Much of that was accomplished by focusing on coordination and cooperation (Do the participants know each other, and how they are supposed to interact?) rather than details of individual performance (Does the city have an adequate cache of equipment and supplies? Do the physicians in the community know how to handle a suspected smallpox case?), which will be assessed in other portions of the evaluation. The participants should be selected by the leaders of the local MMRS. The committee recommends that OEP tell the local MMRS contact only that there will be two simultaneous scenario-driven group discussions and that OEP suggest that he or she should invite representatives of all the major agencies and institutions necessary for an effective response to a mass-casualty terrorism event. In most cases it will not be possible to have all the participating jurisdictions represented, but representatives of local agencies and institutions should not all be from the same jurisdiction. OEP should ask to review the list of invitees before the site visit and should take that opportunity to suggest additions that might be crucial to the discussion. Before the site visit OEP should also attempt to identify some potential critics of the local system, with or without the aid of the local MMRS leaders, and invite them to participate as well. SUMMARY The survey described in the previous chapter provides one tool for assessing the effectiveness of the MMRS program, namely, a survey soliciting the opinions of the communities themselves. This chapter comple-

OCR for page 113
Preparing for Terrorism: Tools for Evaluating the Metropolitan Medical Response System Program ments that approach by presenting the committee’s recommendations for an independent and systematic assessment of the response capabilities of the large metropolitan areas that have or will participate in the MMRS program. Several important assumptions or principles underlie these recommendations: Evaluation should be part of a continuous learning and continuous quality improvement program, not a one-time snapshot. This implies a continuing relationship between the communities and their evaluators that includes financial as well as technical and educational support. “Preparedness” is a meaningless abstract concept without a specific threat; it should be seen as a process rather than a state. Preparedness requires not only numerous specific capabilities, typically the responsibilities of independent offices, agencies, and institutions, but also seamless coordination of those capabilities into a coherent response. The former may be envisioned as the teeth of a comb, the latter as the base or backbone of the comb. Information and the ability to acquire, process, and appropriately distribute it to essential sites and personnel are central to the effective management of critical incidents including terrorism in its many forms. Evaluation is an exercise designed to guide distribution of local, state, and federal resources. Evaluations should be valued and understood as an opportunity for local communities to determine the areas in need of improvement and support rather than as a test of communities’ self-reliance. A relatively small subset of the nearly 500 preparedness indicators identified in the Phase I report (Institute of Medicine, 2001) can be used to identify critical areas in need of improvement for a given community. A set of 23 essential capabilities needed for an effective response to CBR terrorism was presented and used to guide the selection of a subset of preparedness indicators for use in a formal evaluation program. For each of those indicators, the committee then provided its opinion on what would constitute acceptable evidence of preparedness (preparedness criteria). The chapter concludes with the committee’s recommendations on methods for gathering that evidence. Evaluations by OEP should be multilevel processes that include (1) periodic review of documents and records, (2) observation of community-initiated exercises and drills, and (3) an on-site assessment. The committee views the on-site assessment as constituting both interviews with individuals about specific capabilities and a scenario-driven group interaction focused on interagency and institutional cooperation and coordination.