Study on the Long-term Retention of Selected Scientific and Technical Records of the Federal Government Working Papers (1995) / Chapter Skim
Currently Skimming:
III. REPORT OF THE ATMOSPHERIC SCIENCES DATA PANEL
III. REPORT OF THE ATMOSPHERIC SCIENCES DATA PANEL
Pages 52-85
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 52... ...
However, the atmospheric sciences are both blessed and plagued by perhaps the largest data sets of any scientific discipline. Some of the small, but important, data sets from the atmospheric sciences consist of some of the longest time series in any science of data acquired contemporaneously with the events measured (in contrast to evidence of past events)
|
From page 53... ...
Depending on how one defines a single data set, the atmospheric sciences have perhaps 2000 to 6000 identifiable data sets. Throughout the 1960s and 1970s, atmospheric data available for archiving were accumulating at a rate of about 2 terabytes/year.
|
From page 54... ...
Indeed, the handling of all earth science data must be seriously examined from a broad, long-term perspective to assure sensible retention and data management and retention. 2 NATURE OF ATMOSPHERIC SCIENCES DATA Atmospheric sciences data sets are diverse.
|
From page 55... ...
The data rates of traditional types of measurements will continue to grow, but only slowly, as more of our planet becomes instrumented in the course of economic development. Traditional meteorological data consist of measurements in the atmosphere of such quantities as temperature, pressure, water vapor content, wind, precipitation, and cloud cover.
|
From page 56... ...
surface Daily, now 9,000 stations 1900-1993 9415 GB Selected Analyses (mostly global) Main National Meteorological Two times per day, 1945-1993 4850 GB Center analyses increasing at 4 GB/year National Meteorological Four times per day, 1990-1993 458 GB Center advanced analyses increasing at 19 GB/year National Center for Atmospheric Thirty-eight data sets 8 GB Research's ocean observations and analyses European Center for Medium Range Four times per day, 1985-1993 976 GB Weather Forecasting advanced increasing at 8 GB/year analyses Selected Satellites NOAA geostationary satellites Half-hour, visible and infrared 1978-1993 16130 TB NOAA polar orbiting satellites 1978-1993 15 Sounders (IIROS Operational 15720 GB Vertical Sounder)
|
From page 57... ...
humidity, winds Aircraft reports Temperature, 45 Over 1 million observations winds per year are obtained. Ships of opportunity varies 3 hours 140 Snowpack Telemetry 560 Snow water 1 day 15 Run by the Soil Conservation (SNOTEL)
|
From page 58... ...
Though the atmospheric sciences rely heavily on data from many nations, international data sharing has become a contentious issue. A few nations are now attempting to recover the expense of acquiring some atmospheric data through the sale of that information at prices significantly greater than copying and dissemination costs.
|
From page 59... ...
Overlapping time series from different platforms sometimes allow cross calibration and provide some continuity. The large data volumes, the continual change in data processing algorithms, and the continual change in instrument behavior, all raise important issues relevant to the requirements for the long-term archiving of data obtained from space-based platforms.
|
From page 60... ...
Other data are collected less than regularly due to budgetary constraints, such as from rocketsondes into the upper atmosphere and electric and magnetic field measurements. Of fairly direct relevance to the atmospheric sciences are measurements of the earth's water, snow, and ice.
|
From page 61... ...
Field Programs There have been many field programs to collect extensive data for the study of problems in the atmospheric sciences. Programs have emphasized topics such as tropical rain systems, synoptic meteorology of the globe for a whole year, hail formation, the radiative effects of clouds, interannual-climate variations, and many others.
|
From page 62... ...
A small number of very prominent data sets have been made available from data centers or over computer networks. Laboratory Data Laboratory data relevant primarily to the atmospheric sciences are fairly limited.
|
From page 63... ...
Most of the uses discussed above do not need data covering more than a few decades. However, several uses need the longest time series that can be provided.
|
From page 64... ...
NCDC is also World Data Center-A for meteorology. The NCDC collection contains records for most of the traditional meteorological observations listed in Table III.2.
|
From page 65... ...
Additional information on NSIDC is provided in the Geoscience Data Panel report. The National Geophysical Data Center (NGDC)
|
From page 66... ...
The panel finds that foreign weather and other environmental data acquired by the military have great informational value and recommends that ETAC's holdings of foreign data should be considered federal records. As a further consequence, DOD should remain aware of which of its atmospheric data holdings are not duplicated by other agencies (usually NOAA)
|
From page 67... ...
The panel suggests that for all scientific projects funded by federal extramural contracts or grants, the funding agencies should, on a case-by-case basis, evaluate whether the resulting data would be useful as a national resource. If so, the funding contract or grant should require the submission of the resulting data to some federal agency for archiving as federal records.
|
From page 68... ...
. The panel recommends that for projects that involve the collection of scientific data, all agencies (especially NOAA for projects involving the atmospheric sciences)
|
From page 69... ...
The panel suggests that all archived data sets include multiple levels of description of a data set, aimed at users from the novice to the specialist. The report of the Ocean Sciences Data Panel has an extensive discussion of metadata needs and issues.
|
From page 70... ...
Though World Data Centers have been excellent at providing access for scientists to many important data sets, not all of them have stable funding and personnel. The panel suggests that the United States should not necessarily rely on World Data Centers as stable, indefinite archives, especially for data that might be needed during times of national emergency.
|
From page 71... ...
The panel recommends that agencies maintain lists of backup copies of important data sets held by other entities and should check regularly the status of these holdings by other organizations, whether those entities are other government agencies, outside research organizations, commercial enterprises, or foreign data centers. Furthermore agencies should exchange brief memoranda with other organizations providing backup that list the data sets held in duplicate and agreeing to notify the other party before discarding one's own copy of the data.
|
From page 72... ...
In view of the lessons learned from the historical uses of weather satellite data, increasing use of environmental data by numerous groups, and the rapidly increasing volume of the "raw" (level-1 ) data set, the panel suggests that a "staged" or hierarchical data processing approach be adopted for all very large data sets, such as satellite and radar data, shortly after collection.
|
From page 73... ...
In the atmospheric sciences, the total data volume is driven strongly by the volume from a few sensors. It is not easy to cope with the large data flows, but it is possible to use appropriate technology both to save the data and control costs.
|
From page 74... ...
The panel believes that attempts to use best-quality technology, even for huge data sets, could have the perverse effect of forcing people to avoid saving the data. The panel suggests that it is better to save large high-resolution data sets on low-quality media, and accept a little loss, than not to archive those data.
|
From page 75... ...
Historical Climate Data Set. These "historical" data sets have the most stringent requirements for data quality and continuity.
|
From page 76... ...
Priorities Although the panel has been unable to develop hard-and-fast rules for retention decisions, it has been able to agree on some priorities for making those decisions. The panel suggests that in determining the priorities for longterm archiving, consideration should rely most heavily on: the unusualness or rarity of a data set, the level of quality control, the availability of metadata, the value of the data for long time series, · a preference for original data over processed data,
|
From page 77... ...
On the other hand, there is sufficient value in the data sets of model initializations for NCDC to store these data sets for decades, or until improved algorithms have been run with the original data. There is a tendency in the atmospheric sciences to think of all data as being potentially part of an eventual long time series and not being subject to recreation.
|
From page 78... ...
The panel recommends that strong efforts should be made to archive indefinitely all in situ atmospheric science data from outside the United States that federal agencies (including the military) obtain in the course of their activities, even if it is possible to construe these data as not otherwise being federal records.
|
From page 79... ...
s INSTITUTIONAL ARRANGEMENTS AND NATIONAL RESPONSIBILITIES NARA has the general responsibility for overseeing the disposition of all federal records and retaining all federal records worthy of indefinite archiving. This is a daunting responsibility, given the magnitude of the federal
|
From page 80... ...
Custody of Data Following passage of the Federal Records Act of 1950, the National Archives and Congress recognized the appropriateness of the National Weather Records Center of the U.S. Weather Bureau as the appropriate center for operational archiving of weather records.
|
From page 81... ...
Some infrequently used records are physically housed at the Federal Records Center in East Point, Georgia, under a Memorandum of Agreement between NCDC and NARA. These are considered temporary records, mostly manuscripts awaiting conversion to microform.
|
From page 82... ...
Advising and Information Sharing With the exception of some staff members at the large data centers, many federal employees (including government scientists) and most non-government scientists are unaware of the requirements of the Federal Records Act.
|
From page 83... ...
Nevertheless, it is likely that data gathering always will have greater attractiveness than data preservation (NRC, 1995~. 6 SUMMARY The increased attention that long time series of data are receiving in the earth sciences in general, and in the atmospheric sciences in particular, is a very encouraging trend.
|
From page 84... ...
A study should be undertaken to evaluate the tradeoffs and problems of archiving for the long term only a sample of our largest data sets in the atmospheric sciences, with an eye toward finding acceptable sampling techniques. A great difficulty in the funding of archives is deciding who pays for long-term retention and access.
|
From page 85... ...
ACKNOWLEDGMENTS The panel has relied heavily on the varied and rather extensive experience of its members in dealing with atmospheric data, as well as on the numerous studies conducted over the last two decades- studies that are referred to in this report and in which some of the members participated. However, the panel wishes to acknowledge the diverse and substantial inputs provided by the following individuals, marry of whose observations, analyses, and recommendations have been incorporated into this report: Larry Baume of NARA, Thomas Boden of the Carbon Dioxide Information Analysis Center, Dean Bundy of the Naval Research Laboratory, Donald Collins of the National Aeronautics and Space Ads stration, Richard Davis of the National Climatic Data Center, P
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.