Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
3 Evaluation Requirements for Weather Modification Over the years the overriding critical issue for nearly all weather modification research and operational activities has been the need for evaluation and validation of the results. This Committee agrees with flee views stated in many earlier assessments that objectivity' repeatability, and predictability are primary requirements in weather modification research, as well as independent confirmation with strong physical and statistical evidence. In recent years these has beets some improvement in the evaluation and validation of cloud-seeding activities (e.g., more emphasis on randomization and double-blind studies), but these evaluation efforts have not been sufficient to make a clear case for supporting standard methodologies or for achieving predictable results. The challenge for the scientific community is to develop acceptable evaluation criteria to ensure float future research and operational programs build a solid scientific foundation for fi~rthe~ advances. This chapter examines issues related to designing and evaluating weather modification experiments and commercial seeding operations. PHYSICAL EVALUATION The interpretation of observations in the light of established theory and tile development of new theory based on laboratory experiments and observations in the atmosphere are sometimes called physical evaluation. A complete physical-dynamical numerical model of a cloud system (with and without seeding) would be the ideal version of a physical evaluation If meteorologists had the skill to make perfect forecasts, they could estimate seeding effects by simply comparing test results with predictions. But such forecasting skills would requite a complete physical-dynamical model of the relevant cloud systems, as well as a measurement system capable of establishing initial conditions for the model. Neither of these exists nor are they likely to exist in the foreseeable future. In considering the role of weather modification in the field of atmospheric science, it is important to emphasize that many of the uncertainties limiting an understanding of the physics and dynamics of seeded clouds are the same as those that limit quantitative precipitation forecasting in weather forecast models and cloud pet ameterizations in climate models. 39
4() CRITICA L iL8~:Z.JES fair TT'EA TI-lER AIODIFICA TION REtSF-ARCI! An example of a physical evaluation can be found in early weather modification experiments that involved dropping dry-ice pellets into stratus clouds and observing the transition of supercooled drops into masses of ice crystals in the time and locations predicted by laboratory studies and theory. Because the stratus was unifUrn~ over large areas and stable over long time periods (relative to the time required far conducting the experiment)' and because the result could be replicated as often as desired, there was no need for elaborate statistical studies to establish a cause-and-effect relationship between tile seeding and the subsequent develo~n~ent of ice crystals. This example, however, is deceptively simple. Most cloud-seeding ex,veri~nents have not resulted in responses as clear cut and repeatable as that of dry-ice seeding of supercooled stratus Often the cloud systems of interest are lily variable in space and time and this variability is poorly quantified. Convective cloud regions suitable for seeding }save unknown lifetimes and allay be interspersed svith regions where seeding would be ineffective. Thus far we are unable to trace the physical effects from the point of seeding to the end product of rain on the ground. Even our ability to measure the amount of rain reaching the surface leaves much to be desired, although recent advances in radar technology (described in Chapter 4) should lead to better measurement of rainfall. Due to such limitations, cloud scientists have had loo alternative but to turn to statistical evaluations in their efforts to verify seeding effects. STATISTICAL EVALUATION To have reasonable confidence in the results of seeding experiments they must be carefully designed, conducted, and analyzed with the best techniques available. The goal is to minimize uncertainties resulting from the large variability in natural weather systems, from our incomplete knowledge of the physical processes involved, frown our limited ability to measure the relevant meteorological variables and to target seeding agents, and from our inability to replicate experiments (in the strictest sense of the word). Assessments of seeding effects most often consist of comparisons of the amount of precipitation (e.g., rain) measured in a target area with that from a control area. Many of these Compaq isons, especially in the early days of seeding, did not involve randomization. The target and control areas often were the same fixed geographical area, and comparisons were between measurements made during the seeding period and those from a period without seeding. Alternatively? the control area might be a geographically fixed area adjacent to (and meteorologically similar to) the target area. In this case, comparisons are made between measurements from the two areas during the same time periods. In either ofthese designs the comparisons are usually discounted because there is no way to allow for biases arising from temporal or spatial trends that may have been present during the trial period. A more statistically robust design, known as a cross-ove~, uses two similar fixed areas. During each test case ogle area is selected for treatment through a random process while the other serves as the control. it has long been recognized that experimental proof of cause-and-eff~ct relationships (as opposed to chance occurrence) requires randomization and replication (Fisher, 1958), especially when the test pool is highly variable as in the case of weather
EFTA L {JA TlOiV REQUIREI\lE.YTS FOR T7F'F,A TI-lER NIODIFICA TlON, 4 1 systems. A number of randomized seeding experiments have been designed and conducted with the aim of confirming a particular seeding effect. These experiments have provided a large fraction of our scientific data on clouds and storms, but most did not provide evidence sufficient to reject the null hypothesis of no seeding effect. A conclusion commonly reached in these experiments was that "there were indications of seeding effects based ore physical measurements' but the data were not sufficient to reach statistical conclusions." Generally the suggested physical evidence tor seeding effects was deduced frown after-the-fact examination of the data. Front the many kinds of measurements obtained certain ones may be selected because they appear to be associated with a useful seeding effect, perhaps in a particular partition of flee data. The scientist then postulates a mechanism whereby the supposed effect might be linked to the treatment. Regrettably those postulates have not been verified by further experimentation. Statisticians working with meteorologists have developed a range of design and analysis techniques for assessing seeding experiments. In addition to randomization and replication, a well-designed weather modification experiment Clay include pre-screening or blocking to reduce the variance ire the test grOUp7 use of cova~iates, alternating target and control areas (c~oss-over design), and re-randomization as a means of coping with internal variance and small sample sizes Classical hypothesis testing often is replaced by a comprel~er~sive data analysis in which all of flee measured variables are brought to bear on the question of seeding effects (Gabriel t 979; Flueck, 19711. Another relatively new statistical method that may }provide even better evaluation capabilities is the Bayesian technique, which can explicitly account for sources of uncertainty and complicated spatial and temporal dependencies (Appendix B). This technique could have maj or impacts on weather modi fication research if uti 1 ized. 1 ' 1 ~ . . . . ~ . . , . .. ~ . . . . Because of the significant natul al variability in cloud systems, seeding experiments must acquire large numbers of experimental units if a relatively small seeding effect is to be distinguished from chance variations. This has meant long and expensive experiments. Protracted experiments are more vulnerable to secular changes in environmental factors (e.g., weather, land use, background aerosols), many of which can be handled by proper randomization (at least in principled. For instance, it would have . . ~ . . ~ . . , , . ~ ~ . . ,~, ~ , - '. r. taken over 50 years to carry out a full statistical evaluation of the effects of seeding on hurricanes using ] 970s technology (Simpson et al., 2002~. The ACWC introduced the concept of exploratory arid confirmatory experiments to differentiate between searching for possible seeding effects and formal testing of a postulated effect. Statistics can be used in an exploratory mariner to guide Understanding of the important physical processes in a conceptual model For instance, in recent hydroscopic seeding experiments (i.e.? the South African, Mexican, and Thailand experiments described in Appendix A) statistical analyses indicated increases in rainfall, but they appeared later in time than anticipated and did not conform to the original hypothesis. Dynamical effects, which were riot included in the original hypotheses, were invoked to explain the results. The statistical analyses thus led to the development of new hypotheses to explain the experimental results. Some may argue that a single test variable is necessary to guard against multiplicity and to provide an unambiguous proof of concept. However' data from cloud-
~2 CRITICA L I~SS(JES I.\ THEA TI-IER AIODIFI(.-A TION RE,SEARCl-I seeding experiments are highly variable, and this reduces the power of a single test to detect differences. To fully consider and evaluate the myriad of variables in weather modification experiments, multivariatc statistical process models that exhibit spatial and temporal dependence are much better suited (Appendix B). Statistics can be used not only as a tool to test proof of concept, but also as a tool ~~ discovery (a mathematical "magnifying glass?'). The advances in statistical sciences described in Appendix B leave not yet been fully applied in weather modification research. Application of these methods' together with the advances in measurement technology and modeling, promises improvements both ir1 verification and in our physical understanding of the processes involved. Appendix B provides an in-depth scientific discussion of the current methods available in statistical science with direct application to weather modification. In the sections that follow a mole genera] discussion of evaluation requirements in weather modification is presented. MEASUREMENT UNCERTAINTIES Even though the classical methodologies of testing cloud seeding are well established, several kinds of difficulties are encountered in practice. Tl~e objective of assessing the results of a cloud-seeding experiment is to establish whether the test variate, such as the total lain in a target area under treatment, is different than it would have been with no treatment. Obviously, one must then be able to measure the test variate with sufficient accuracy to separate the effects of treatment f om natural variability. This has been a major problem in cloud-seeding experimentation. For instance, experiments aimed at increasing rainfall typically have used networks of surface-~ain gauges as their measurement system Rain gauges give a fairly accurate measurement of rain at the point of the gauge, but rain is highly variable in space and time, especially in convective weather situations. The frequency distribution of storm rainfall amounts is highly skewed, with a large number of small events interspersed with a small number of large events that account for most of the total lain. Title the density of rain gauges normally attainable, and integration over periods of hours? area- average rain amounts have large errors, especially in convective situations. Radar is being used more frequently for measuring rain, with the advantage of much better spatial coverage and temporal resolution. But this introduces another variable, namely, the relation between the measured tadar parameter and rainfall at the surface, which depends on the drop-size distribution, which may be affected by seeding. Other direct and indirect measures that have been used for assessing seeding trials, such as hail-fall energy and crop damage estimates, also introduce additional layers of variability that must be accounted for. UNCERTAINTIES IN DEFINING AND TRACKING THE TARGET In many cloud-seeding experiments the experimental units are elusive, hard to define, and difficult to follow in time. In fact, to see a convective cloud as a single entity is an illusion Clouds are transitory, always evolving and mixing internally and with their
E f A L (JA TION REQlfIREtt1ENTS FOR T7FrEA Tl-TER N1ODIFICA TiON 43 environment. These basic properties of clouds make it difficult to keep track of seeded units and to replicate the treatment in successive trials. At the same time? however, this inherent mixing within the atmosphere plays an essential role in most seeding experiments. In the immediate vicinity of the release point from seeding devices the concentration of seeding materials is much too high Or effective cloud treatment. Operators depend on atmospheric Nixing to dilute the seeding material before it reaches the target area. Further mixing then reduces the concentration of seeding materials and Nay reduce it to the level where it becomes ineffective. No two clouds are identical, and clouds are riot independent of one another. A limited number of experiments have found that traces materials released into the sub- cloud updraft of a developing convective cloud were subsequently found in the rain coming from neighboring clouds, thus suggesting some degree of interaction. The unmount of cloud interaction probably decreases with separation in space and tithe. The degree of dependence between different clouds on the same day, or clouds in the same area and air mass on different days, is not measurable and thus hard to allow for in assessing seeding trials. This issue is often simply ignored in many seeding studies. However, recent advances in this area integrating several observational tools could help to address these issues (Mueller et al., 2003~. A variety of tracking methods and software packages are used to evaluate the results of seeding activities For example, the South African and Mexican hyg~oscopic seeding experiments (WMO, 2000) used the sto~ms-based Thunderstorm Identification Tracking Analysis and Nowcasting (TITAN) tracking software to evaluate their radar- based r esults. the hygt oscopic seeding experiments in Thailand (Silverman and Sukarnjanaset, 2000) used a va~iable-radius floating target that moved with the mean radar echo motion. The recent glaciogenic experiments in Texas (Rosenfeld and Woodley, 1993) arid Thailand (Woodley et al, 2003) used a hierarchy of radat-tracked cells imbedded in fed-radius floating targets that are moved with the mean-cell motion. , , , UNCERTAINTIES lN REACHING THE TARGET When ice-forn~ing agents are released directly into the top of a supercooled stratus cloud, there is little question whether it reaches ~ susceptible region of cloud. When the seeding agent is released directly into the updraft under a convective cloud it will become part of the updraft and presumably will be carried to a level where it can be effective. In the case of atea-wide sub-cloud seeding and orographic seeding, the agent usually is released upwind of the target. Whether it reaches the intended target, and if so in what amounts will depend on the winds and turbulence between the release point and the target. In some contexts the means for- measuring and forecasting these winds in real- time is very limited and thus is another source of uncertainty. Some seeding particles from go ound-based generators could be scavenged by snow and ice and therefore diminish the effects of seeding (Warbu~ton et al., 19951. For all of these reasons the targeting and mixing of the seeding material through a cloud remains lushly uncertain. However, with new high-resolution mesoscale numerical models and remote sensors,
CR JTI(',4 L [~SSlJES IN [YEA TI -TER A JODIFICA TION REt5~-A RCI-J new opportunities exist to address these issues, especially in winter orographic situations Becloud and cloud-top seeding introduces similar uncertainties but could potentially be addressed with new modeling and observational tools. As discussed in Chapter 4 the use of cliff fibers or gaseous tracers may be a particularly good strategy for tracking the dispersal of seeding material and the resulting cloud enacts. ASSESSING THE AREA AFFECTED The areas affected by cloud seeding remain an open questions. In after-the-fact analyses several rain enhancement projects have reported evidence for physical effects outside the area or timing originally designated as the target, or beyond the time interval when seeding effects were anticipated. For example, in recent large particle hydroscopic seeding trials involving war-base convective clouds in Thailand and Texas, increases in rain were reported 3 to 12 hours after seeding was conducted' well beyond the time at which direct effects of seeding were expected and possibly outside the target area. In Project Whitetop the seeding appears to have decreased rain in the area immediately downwind of the seeding release line. This was followed by apparent rainfall increases well downwind in space and time. Does this mean that the scientists misjudged where seeding materials were actually reaching receptive cloud conditions or does it mean that the primary effects of seeding were fallowed by secondary effects well beyond the original target? Such secondary effects could occur, for instance, if seeding materials become entrained in a downdraft and then are carried outward into the updraft of other clouds. In the case of the hydroscopic seeding experiments the postulated dynamic effects due to micronhYsical and dynamical interactions in the cloud and sub-cloud region and _O ~ ~ . ~ ~ , .,, ,, . , it, ,, . , .. . . . ~ .. . . with the environment could result In longet-Jived or progeny clouds. Another related uncertainty in seeding convective systems is whether a positive effect on some individual clouds (or cloud complexes) will aggregate to result in increased area rainfall. An associated question addressed in Appendix A and Box l .1 is that of throbbing Peter to pay Paul." Debates about the effects of seeding beyond the target area point to the fact that weather modification can be viewed as more than just a means to increase local precipitation. Rather, it can be viewed as a means to alter natural hydrological cycles by increasing the number of times that atmospheric water is recycled at the Earth7s surface. As mole is learned about the globe] water balance and as new tools enable the cloud scientist to better understand clouds and their response to seeding, the question of extended area affects likely will become better defined and understood.
