Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
A Framework for Indicator Selection Ecological indicators that describe the state of the nation's eco- systems and command broad and deep attention by the public and decision makers have not as yet been developed. The failure to achieve compelling nationwide ecological indicators, despite consider- able effort, results from the complexity of ecological systems, their vari- ability in space and time, and the great variety of human interactions with natural and modified ecosystems. But inadequate attention to the criteria that should guide development and use of all indicators has also contrib- uted to the failure to achieve successful nationwide indicators. Many existing ecological indicators are applicable to only limited areas, to one type of ecosystem, or to the populations of one or a few species. Such indicators are likely to continue to be useful for their intended purposes, but by their very nature, they cannot serve as nationwide indicators. Some indicators have been less useful than hoped because the mea- sures employed are not clearly linked to underlying ecological processes. As a result, it has been difficult to interpret changes in those indicators. The input data requirements of still other indicators are so complex and extensive that the costs of gathering these data have been unsustainable or have presented barriers to the use of the indicator. For example, there is not enough taxonomic knowledge to use species richness of micro- organisms in soils or algae in aquatic systems as environmental indica- tors, and even if there were, the cost of using them would be very high because the systems would have to be sampled so frequently. To compute some indicators, intrinsically heterogeneous variables 51
52 ECOLOGICAL INDICATORS FOR THE NATION must be combined. Most currently used biotic indicators and indices suffer from problems of ambiguity in index scoring, combinations of unrelated measures, and severe limits on diagnostic abilities (Suter 1993~. Complex issues may surround how such information should be com- bined, and a combined index of heterogeneous variables may be difficult to interpret (Washington 1984, Plafkin et al. 1989~. These types of prob- lems in indicator development and interpretation have plagued scientists and managers for years. CRITERIA FOR EVALUATING INDICATORS To avoid the pitfalls that limited the value of earlier ecological indica- tors and to provide a common framework for developing, describing, and evaluating indicators, the committee developed a general checklist. The checklist can be used to assess the potential importance of a proposed indicator, its properties, its domain of applicability, and its limitations, and thus how the indicator might be used. The entries in this checklist, reviewed over the remainder of this chapter, are general importance; conceptual basis; reliability; temporal and spatial scales of applicability; statistical properties; data requirements; necessary skills of collectors of the data; data quality control, archiving, and access; robustness; inter- national compatibility; and cost-effectiveness. General Importance Does the indicator tell us about changes in the primary ecological and biogeochemical processes described in the committee's conceptual model? Does the indicator tell us something about major environmental changes that affect wide areas? Not all indicators need to be of nationwide or international scope, but because the committee devoted most of its efforts to identifying, propos- ing, and characterizing indicators of general applicability, this criterion assumed major importance in the committee's deliberations. The domains of applicability of indicators of regional or local processes, or those that pertain to particular species of interest, also require attention. The spatial and temporal domain and the array of states and processes that an indica- tor reflects need to be specified and understood for all indicators. Conceptual Basis Is the indicator based on a well-understood and generally accepted conceptual model? Is it based on well-established scientific principles? As we have noted, an indicator is not likely to be useful unless it is
A FRAMEWORK FOR INDICATOR SELECTION 53 based on a conceptual model of the system to which it is applied. The conceptual model provides the rationale for the indicator, suggests how it should be computed, and enables us to understand the features of the indicator and how they change. Without a supporting model, an indicator's meaning and the right approach to interpreting it remain unclear (Landres 1992~. Reliability Has past experience with the indicator demonstrated its reliability? What other evidence exists for its reliability? The best evidence for the reliability of an indicator is, of course, its successful use previously. Nevertheless, all existing indicators should be analyzed retrospectively before assuming that their use should be contin- ued. Inertia may lead to continued use of indicators that should be dis- carded in favor of better ones, or at least modified as a result of experience in their use and interpretation. An indicator that is newly proposed inevitably lacks a historical record of reliability. Nonetheless, if it is based on a well-established scientific theory, and if a retrospective analysis has indicated that it probably would have informed us about important changes in an environmental process or product of concern, its reliability is provisionally established. Still another check on the reliability of indicators is comparing them with already-used indicators that share a similar scientific rationale and employ similar input data. If those indicators have proven to be reliable, the proposed one is likely to be as well. For example, a variety of community-level diversity indicators based on stream invertebrates have been patterned after established indicators also based on the diversity of fishes in streams. Thus, the Invertebrate Community Index is essentially equivalent to the IBI (Plafkin et al. 1989~. Temporal and Spatial Scales of Applicability Does the indicator inform us about national, regional, or local processes and products? Are the changes measured by the indicator likely to be short-term or long-term? Is the indicator sensitive enough to detect important changes but not so sensitive that signals are masked by natural variability? To determine what an indicator indicates, the kinds of data needed to compute it, and how changes in it should be interpreted, the temporal and spatial scales of the processes measured by the indicator need to be clear (Peterson and Parker 1998~. Much about an indicator's relevant spatial scale is shown by the scope of the data used to compute the
54 ECOLOGICAL INDICATORS FOR THE NATION indicator, but temporal scales of applicability are much more difficult to determine, although paleoecological studies may be helpful in doing so. For example, the number of taxa, the lack of good methods for delineating them, and the heterogeneity of community composition at microscopic scales make it infeasible to assess the diversity of microorganisms at present. Over the past decade, ecologists have gained an awareness of the critical importance of scale. Virtually all metrics of ecosystems depend on spatiotemporal scale, so the explicit selection of spatial and temporal scales for indicators is essential. The field of spatial statistics and geo- statistics provides an extensive foundation for the description of spatial patterns and how they change across spatial scales. Foundations for the variety of statistical methods and tests used to assess spatial descriptors are found in Bartlett (1975), Ripley (1981), Cliff and Ord (1981), and espe- cially Cressie (1993~. These methods allow identifying the scale at which a pattern exhibits the least random variation and the scale at which the pattern changes abruptly. Spatial statistics are useful in evaluating alternative scales for indica- tors. All else being equal, the better scale is one at which an indicator exhibits the least stochastic variation and the weakest dependence on small changes in scale. Spatial statistical patterns might also serve as indicators of spatial qualities, such as landscape heterogeneity or habitat diversity. The primary problem with spatial statistics is their require- ment for extensive input data. Except for satellite and aerial imagery, most ecological data sets lack the necessary sample sizes and spatial coverage. Without knowledge of the scaling rules, the scales at which measurements are made cannot be extrapolated to the scales at which indicators are needed. This is an issue that would benefit from a focused research program, using data from a large sampling program such as EPA's Environmental Monitoring and Assessment Program (EMAP). Statistical Properties In the areas of accuracy, sensitivity, precision, and robustness, has the indicator been shown to be good enough to serve its intended purpose? Can the indicator detect signals above the "noise" of normal environ- mental variation? Are its statistical properties understood well enough that changes in its values will have clear and unambiguous meaning? Because ecosystems vary in space and time, indicators of their status and functioning also vary spatially and temporally. In addition, eco- systems are changing today in unprecedented ways. Lack of historical and contemporary data make it difficult to define clearly the nature and extent of these changes (NRC 1986), although paleoecological studies can
A FRAMEWORK FOR INDICATOR SELECTION 55 be useful. Useful indicators should be able to distinguish between "normal" variation and variation that falls outside what is expected given the historical and paleoecological record. Ecological indicators are designed so that changes in their values signal significant changes in ecosystems, changes to which attention should be paid. When a signal must be detected against a variable envi- ronment, which is true in most ecological conditions, it is essential to consider which of two types of errors is more important to avoid (Simberloff 1990~. The first, known as a Type I error, is to conclude that a significant change has occurred when it has not, that is, interpreting normal environmental variability as a real change. The second, or Type II error, is to conclude that there has been no significant change when in fact such change has occurred. When a Type I error is made it may prompt an unnecessary, costly, ineffective, and possibly counterproductive response. When a Type II error is made, a needed response may not be made, which may compound a serious problem. When the needed responses are even- tually initiated, they may be both more costly and less effective in allevi- ating the problem than if they had been undertaken earlier. Each indicator should be evaluated to assess the relative importance of Type I and Type II errors associated with its use. (Statistical issues are considered further in Appendix A.) Data Requirements How much and what kinds of information are necessary to permit reliable estimates of the indicator to be calculated? How many and what kinds of data are required for the indicator to detect a trend? All indicators require input data, but what they require differs dra- matically in nature and extent. Most ecological indicators depend on data gathered by means of long-term monitoring. The challenge is deciding which rates of change to watch, and to determine which of the changes observed are normal and which are not. Often changes in rates can be determined only after long periods of time. The significance of changes over short time periods is often unclear because the record is too short to characterize natural variability in the system. Temporal and spatial varia- tion is often considered something to minimize by clever sampling design, but such variability may demonstrate the most interesting and important features of the system (Kratz et al. 1991~. Ecosystems may change very slowly, in response to factors such as changing climate, soil properties, and evolutionary changes in species; moderately slowly, as vegetation succession occurs and species ranges change; or suddenly, in response to disturbances, either natural (e.g., fire, storm, or disease) or anthropogenic (e.g., acid deposition, wetland drain-
56 ECOLOGICAL INDICATORS FOR THE NATION age, forest clear-cutting, or species extinctions). To develop an indicator that can signal significant ecosystem changes, whether in response to natural or anthropogenic perturbations, it is essential to identify current system states physical, chemical, and biological in both stressed and unstressed ecosystems. Where possible, it is also desirable to establish the past states of the system through historical and paleoecological studies that provide baselines for a program of physical, chemical, and biological monitoring. Those properties of ecosystem structure and functioning that paleoecological studies have identified as significant indicators of change are prime candidates for inclusion in current monitoring programs. Infor- mation from the monitoring program can then be used to devise an indi- cator whose properties can be specified and interpreted. Once an indicator is selected, monitoring must be used to increase knowledge about the likely meaning of changes in the indicator's values. Experimental studies themselves requiring monitoring should be used to determine whether the stress/response relationships suggested by the monitoring program are indeed causal. The use of the indicator may change as additional insights are gained into its behavior and the under- lying processes that cause it to change. Necessary Skills What technical and conceptual skills must the collectors of data for an indicator possess? Does the collection of input data require highly techni- cal, specialized knowledge if the data are to be accurate, or is data collec- tion a relatively straightforward process? An indicator capable of commanding broad attention must be based on data that are accurate and, equally important, perceived by all inter- ested parties to be accurate. Accuracy, both real and perceived, is more likely to be achieved if it is clear how the input data are collected and that the gatherers of the data have skills appropriate to their assigned tasks. Thus, it is desirable that indicators be designed to use input data that are relatively straightforward to gather. Because collection of data for ecological indicators (monitoring) is sometimes perceived by scientists as boring or less interesting and presti- gious than "scientific research" (i.e., hypothesis-driven investigation), it is important to provide incentives for consistent and accurate data collection. Monitoring data can often have great scientific value, but not necessarily to the individuals who collect them. Much scientific research involves the painstaking and repetitive collection of data, but the potential reward of being able to answer a scientific question is often sufficient incentive to lead people to spend long hours and even sleepless nights to collect them. For this reason, as well as for the obvious scientific benefits that would
A FRAMEWORK FOR INDICATOR SELECTION 57 accrue, the committee suggests that monitoring of ecological indicators be coupled with more focused scientific research whenever practical. This could mean taking advantage of long-term research projects (such as the National Science Foundation's Long Term Ecological Research {LTER} sites), or inviting researchers to make use of data collected by a monitor- ing program to test scientific hypotheses. The indicators we have pro- posed embody hypotheses about the functioning of ecosystems. To the degree that such hypotheses can be made explicit in the design of indica- tors, their development and the subsequent monitoring of them should generate a great deal of valuable scientific information. Other incentives might also help to achieve the goal of obtaining consistent and accurate monitoring data. Robustness For our purposes here, we define robustness in a nonstatistical sense, as an indicator's ability to yield reliable and useful numbers in the face of external perturbations. In other words, is the indicator relatively insensi- tive to expected sources of interference? Are technological changes likely to render the indicator irrelevant or of limited value? Can time series of measurements be continued in compatible form when measurement tech- nologies change? Indicators are influenced both by changes in the systems monitored and by changes in technology. During the life of an indicator, scientific advances are likely to improve understanding of the system's dynamics and hence the ability to interpret changes in the indicator. The expanded knowledge may also suggest that additional data should be included as inputs to the indicator, or that data previously collected are less relevant than they formerly appeared, and perhaps their use should be discontin- ued. In addition, technological advances may enable measurements to be made that are currently impossible. Other measurements may become much easier. Thus, the input data to indicators are likely to change. Such technologically driven changes are, of course, welcome. To con- tinue to gather data by outdated methods is undesirable. Nevertheless, because long-term data sets are essential for detecting most environmen- tal trends, technological changes must be incorporated into monitoring programs in ways that do not destroy the continuity of the data sets or render consistent interpretation of the changes impossible. As pointed out in Chapter 2, cross-calibration of measurements is especially impor- tant for remotely sensed data.
58 ECOLOGICAL INDICATORS FOR THE NATION International Compatibility Is the indicator compatible with indicators being developed by other nations and international groups? Not all indicators used in the United States, especially those relating to specific regions, ecosystems, or species, need to be compatible with indicators developed and used in other nations. However, national-level indicators signal changes that are likely to transcend national boundaries. Effective responses to these changes may require international action. If the signals that trigger actions are not mutually interpretable to the affected nations, appropriate responses are certain to be more difficult to mount. Costs, Benefits, and Cost-Effectiveness Costs and benefits associated with proposed indicators are important because resources for monitoring are limited and should be used effi- ciently. The costs of developing and monitoring an indicator, which can continue to accrue as the indicator is used and refined and as new data and technologies develop, can be estimated objectively. The benefits- the value of the information obtained are more difficult to estimate. They include both its contribution to scientific progress and to improve- ments in societal decisions, and they also continue to accrue over time. The greater the contributions by an indicator, the higher the costs that can be justified in developing and implementing it. For example, developing and monitoring a national land-use indicator will be costly, but the com- mittee considers that the information will be of great value and will be an essential component of other indicators. Indicators can be judged based on several criteria, an important one of which is cost-effectiveness. If one assumes that the information an indicator yields is essential, can it be obtained for less cost in another way? If so, the indicator is not cost-effective (Landres 1992~. Another criterion is more restrictive: Is the value of the information to be obtained greater than the cost of obtaining it? This is the positive-net-social- benefits test, which is difficult to apply because it requires dollar (or some proxy) measurement of the societal value of the information. The value of the information was the committee's first consideration in every indi- cator we recommend. . INFORMATION HANDLING AND CALIBRATION Although tremendous strides have been made in remote sensing over the past six years, the use of this information for ecological analyses is still in its infancy. For many years, the technological challenges of handling
A FRAMEWORK FOR INDICATOR SELECTION 59 and processing the data were so great that only the most sophisticated laboratories could use these data. Because rapid improvement, in both cost and performance, of computer hardware and software is now remov- ing many technical impediments to the use of remotely sensed data, it is increasingly important to pay attention to the care, maintenance, and accessibility of data archives, and to the intercalibration of the remote- sensing instruments themselves. The processes and conditions for which the committee recommends indicators operate at multiple spatial and temporal scales. Instruments that are used to measure temporal and spatial variation must be cali- brated carefully to ensure confidence in ascribing changes in measure- ments to the ecosystems being monitored, rather than to the instruments themselves. Although simple to stipulate, achieving calibration precise enough for quantitative scientific measurements is very difficult. Delicate hardware that has been calibrated and tested on the ground, for example, is subjected to tremendous vibrational stresses during launch and then to the thermal and radiational stresses of low Earth orbit. Maintaining instrument calibration is a nontrivial task under these circumstances, but it can be achieved, as the Landsat data record shows. For data sets that must last longer than the lifetime of any one instru- ment, it is equally important to ensure that successive instruments are flown for a period of overlap. As both the hardware and software of instruments evolve, one must be able to identify and quantify degrada- tion of instrument performance and offsets due to changing satellite orbital geometry or new technology, and correct for them before further analyses are attempted. DATA QUALITY CONTROL, ARCHIVING, AND ASSIGNMENT OF RESPONSIBILITIES Before any indicator is adopted, substantial thought and effort need to be given to issues of data quality control, data archiving, and data access. The integrity of time series of information is vital because indi- vidual measurements acquire value only when they are compared with the same measurements from other similar ecosystems or from the same ecosystem at other times. The data on which ecological indicators are based must be archived and available to a wide range of interested parties if the indicator is to be accepted and used to guide policy. Before any indicator is designed and used, its archival requirements must be considered carefully and accom- modated. Issues that need attention include the following:
60 ECOLOGICAL INDICATORS FOR THE NATION · How and by whom will quality control over input data be ensured? · Who are potential users of the data and how can their needs be met? · How can the data be used to improve the models on which the indicator is based? · How can the archival system best accommodate technological changes in both data collection and archiving methods? · Who will coordinate and manage the archives? · How can the system respond to complex user queries that may require new analyses and interpretation of existing data? · How will the data storage systems be integrated with other archi- val systems of federal, state, and local governments? Data Quality Control The indicators the committee recommends are calculated from exten- sive underlying empirical data collected by a variety of monitoring pro- grams. Each of these programs has some capacity to ensure that their data are of high quality, that they are archived appropriately, and that they are accessible. No indicator of environmental quality is reliable unless the under- lying data that are used to construct or calculate it are accurate. No amount of attention to data quality during the archiving and computa- tional phases can substitute for the quality of the input data. In this critical sense, the ultimate responsibility for data quality must lie with the investigators who collect them. Therefore, a successful monitoring system must ensure that there are sufficient incentives in place for participating investigators to maintain quality checks on their data. Clear documentation of sampling and analytical methods is necessary if future investigators are to understand exactly how each indicator was derived. Therefore, the original investigators must document their methods carefully enough so that someone not associated with the original data collection can reproduce the original sampling or analytical protocols. This requirement is particularly important as methods and instrumenta- tion change, so that data from early parts of the time series are quantifiably comparable to data from later parts of the same time series. Individual investigators must clearly record changes in their methods, and docu- ment the influence of those changes on the measurements. Data Archiving A monitoring system to track ecological indicators requires archiving capabilities that provide interested parties access to the data. To design
A FRAMEWORK FOR INDICATOR SELECTION 61 an archiving system, it must be clear whether "raw" data are to be archived, or only derived quantities such as the indicators themselves. For indicators that are direct representations of environmental samples, the archiving job is simple: the archive simply needs to save a record of the measurements, such as phosphorus concentrations. A more difficult question is to what degree the physical samples themselves (e.g., soils, water, or plant, animal, and microbial specimens) should be archived. In general, the minimum number of physical samples saved should ensure the ability to recalibrate the entire data set, should that become necessary because of changes in sampling or analytical technolo- gies (so-called technology drift). The costs of preserving physical samples in forms that do not decay or otherwise change must be weighed against the opportunity cost of not being able to recalibrate a data set with improved or modified measurement techniques. Indicators of net ecosystem productivity or net primary productivity for any substantially large area are computed using some form of remotely sensed data combined with model simulations. The archive must encom- pass procedures to preserve not only the original data, but also the models that have been used to interpret or extrapolate from them. Governments are likely to be the source of most remotely sensed data for the next decade or so. Substantial provisions have already been made to ensure that the original digital data are archived in easily accessible and afford- able forms. Little could be gained by another monitoring system to duplicate the archiving of these data. However, remote-sensing archives do not necessarily save derived products and other analyses computed from the original data, unless they are specifically funded to do so. There- fore, estimates of productivity and similar derived values should also be archived, together with pointers towards the original imagery and a complete description of the models that have been used to derive the estimates. The complete description and availability of the models and their metadata used to derive final indicators are just as important as the avail- ability of the underlying remote-sensing data themselves; otherwise, future comparisons might not compare the same things. The models that have been used to date to calculate such variables as ecosystem produc- tivity and net primary productivity are active research tools, and they too will continue to evolve. Therefore, the archival task is to ensure that there is as much traceability in the models as there is in the measurements. The archive must therefore be robust enough to ensure that the time series of the indicator can be reprocessed as models improve. Multimetric indicators face similar problems because they are calcu- lated from sampling data. If the biological sampling data are part of ongoing research or monitoring efforts that have already made provi-
62 ECOLOGICAL INDICATORS FOR THE NATION signs for archives, there may be little need for additional archiving capa- bilities. However, if the biological data are not otherwise archived, then the national system must make provisions for archiving the original data, and not merely the values of the index itself. Assignment of Responsibilities How should the responsibilities of individual investigators be assigned and monitored? At a minimum, the monitoring systems that contribute data to indicators must include some central function that establishes and maintains appropriate standards for data quality that investigators who make the measurements must meet. The standards themselves do not need to be centrally created and unilaterally handed to investigators. Instead, the investigators themselves can discuss and develop appropriate standards that they agree to uphold. However, the central mechanisms must be adequate to ensure that when data are pro- posed for entry into archives, they satisfy the agreed-on standards. This process differs from the centralized data quality control processes that have often been used in large monitoring and research programs. It envisions the central function as that of a monitor, rather than a con- troller. The responsibility for setting standards, for changing standards when appropriate, and for upholding standards, is the responsibility of the investigators themselves. Implementing this function in the monitoring systems that provide data for indicators will be expensive. Experience with large field pro- grams (e.g., the National Aeronautics and Space Administration's First ISLSCP {International Satellite Land Surface Climatology Project} Field Experiment {FIFE}, the National Aeronautics and Space Administration's Boreal Ecosystem Atmosphere Study {BOREAS}, the National Science Foundation's Long-Term Ecological Research {LTER} ~ suggests that the costs of the data systems that support intensive monitoring efforts can easily amount to 25 percent of the total budget. Nevertheless, the moni- toring of input data is critically important and must be supported at levels that ensure that data quality and compatibility do not decline over time. USE OF THE COMMITTEE FRAMEWORK In the following two chapters, the committee uses the framework described in this chapter to identify a coherent set of indicators that can provide a comprehensive view of the status and trends of the nation's environment. These highly aggregated national-level indicators, which are based on well-established scientific data and models, require exten-
A FRAMEWORK FOR INDICATOR SELECTION 63 sive measurements from all parts of the nation as input. As we show in Chapter 5, if the data are archived in a highly disaggregated form, they can be used for indicators designed for local and regional levels as well.
