Appendix D
National Weather Service Flash Flood Verification Procedures1
HYDROLOGIC VERIFICATION PROCEDURES FOR FLASH FLOOD WARNINGS
The Office of Climate Water and Weather Services (OCWWS) Performance Branch is responsible for the operation and maintenance of the automated flash flood warning (FFW) verification program.
Matching Warnings and Events
All warning data are automatically extracted from the warning products issued to the public. FFWs are issued by county. Since each county specified in a warning represents a separate verification area, a warning covering three counties is counted as three warned areas or three warnings. Events are automatically taken from the final Storm Data reports prepared by the WFOs. Storm Data reports entered as the event type “flash flood” verify an FFW.
For verification purposes, multiple flash flood events in the same county separated by less than 30 minutes are considered duplicates; therefore, only the first entry is made to the event database. This rule has the following exceptions:
1 |
This information is excerpted from the National Weather Service Operations and Services Performance manual 10-1601 entitled “Verification Procedures” (NWS, 2004d). This manual was revised July 20, 2004, after the committee completed its analysis. The revised version includes quality assurance rules, which have been in effect since January 2002 but were inadvertently left out of the prior version, dated January 6, 2003. Because the committee adhered to the January 6, 2003, version, it did not utilize the quality assurance rules in its analysis. In particular, consistent application of Rule 2 would modify some of the entries in Table 7.1. |
-
any event that causes death or injury is included in the event database;
-
any event that causes crop or property damage in excess of $500,000 is included in the event database;
-
an event is not considered a duplicate if it is the only event verifying a warning.
Any event not recorded in the verification database due to the aforementioned duplicate rule still appears in the publication Storm Data.
Warnings and events are recorded in separate databases. Whenever an event occurs in a warned county, the following are recorded: one verified warning and one warned event. One unwarned event is recorded for each event that occurs in a county with no warning. One unverified warning is counted for each warned county that does not experience an event.
Quality Assurance Rules
In an attempt to reduce the impact of erroneous short-fused warnings on customers and, at the same time, more accurately measure the quality of NWS warnings, the OCWWS Performance Branch has developed a set of rules stating how these short-fused warnings are archived.
-
Rule 1: How Warnings are Entered into the Database. All data imported into the warning database are taken directly from the warning. No data are entered into the database from any information other than that represented by the bold-faced parts of the warning sample in Table D.1. Based on this rule, products issued with the improper coding may or may not be imported into the database. Several examples appear on the NWS Verification Web Page.
-
Rule 2: Quality Assurance of Overlapping Warnings. When two warnings for a given county overlap in time, the portion of the earlier warning that overlaps the second warning is removed. The expiration time of the first warning is changed to one minute before the issuance time of the second warning. Several examples appear on the NWS Verification Web Page.
Lead Time
For verification purposes, the definition of the term “event” is given above. The lead time for each flash flood event is computed separately for each county by subtracting the time of warning issuance from the time
TABLE D.1 Flash Flood Warning Sample
|
when the event first occurred in the county. Negative values are converted to zero. If one or more events occur in a county with no warning in effect, each unwarned event is assigned a lead time of zero. Average lead time is computed from all lead times listed in the event database, including zeroes. The percentage of events with lead time greater than zero is also computed.
Display of Verification Statistics
NWS employees access FFW verification statistics through the Stats on Demand feature of the NWS Verification Web Page. Stats on Demand accesses an interactive database that provides verification statistics customized to the user’s request. The user may request data by:
TABLE D.2 Example of Format for Reporting Preliminary Flash Flood Warning Verification Statistics
Preliminary Verification Statistics |
Flash floods |
Number of warnings issued |
|
Number of verified warnings |
|
Number of unverified warnings |
|
Number of events |
|
Number of events with warnings |
|
Number of events without warnings |
|
Average lead time |
|
Probability of detection |
|
False alarm rate |
|
Critical success index |
|
-
one or more dates (select beginning and ending date);
-
one or more counties, WFOs, states, NWS regions, or the contiguous United States.
Backup Mode for Warnings
When a WFO goes into backup mode, FFWs are still sorted by county, so all FFWs issued by the backup office are attributed to the primary WFO.
Preliminary Flash Flood Warning Reports
The regional headquarters will report to the OCWWS Performance Branch no later than the 14th of each month preliminary flash flood warning verification statistics for the previous month. When the 14th of the month falls on a weekend or holiday, the statistics are due the last business day prior to the 14th. Present the statistics in the format given in Table D.2. The OCWWS Performance Branch subsequently collates the regional data into a national summary.