22

Evaluating Data Management Productivity and Performance in Government:
The View from the Trenches

Thomas Mace

     First of all, what is the Data Management Working Group (DMWG) for the U.S. Global Change Research Programs, and what does it do? The DMWG has two functions. Our traditional role has been to report to the Subcommittee on Global Change Research under the Committee on Environment and Natural Resources (CENR) as the interagency coordinating body for data management across the participating U.S. federal agencies. We also have a subgroup called the Global Change Data and Information System (GCDIS), and it forms the core activity in support of the U.S. Global Change Research Program. Our second function is to provide interagency coordination for data and information across all the CENR subgroups.

     A bit of history: This group has been around since 1987 when it started out as the Interagency Working Group on Data Management for Global Change. We had two senior management groups that met roughly quarterly, along with a group of us that were contacts, who eventually became the Data Management Working Group. After a number of years the senior managers decided they would just review us perhaps once a year, and let us run on our own.

     In 1993, there was a more formalized process, and we went from being an ad hoc committee to a formal subgroup of the Subcommittee on Global Change Research (SGCR). In 1994, we also became attached to the Task Force on Observations and Data Management, and really began working on data management for the entire CENR.

     Who are the principal contacts of the DMWG? In addition to the SGCR agencies, approximately 30 to 50 people attend the meetings. (We meet roughly monthly.) So the agencies are represented pretty much all across the board. For example, we have representatives that come from the Smithsonian and from the Tennessee Valley Authority at times.

     There are several key elements of the DMWG. First, and probably the most important point, most of the action occurs in agency data centers. These are staffed by professional groups of people from places like Oak Ridge National Laboratory, the National Computing Center at the Environmental Protection Agency (EPA), the National Aeronautics and Space Administration's (NASA) Distributed Active Archive Centers, and the U.S. Geological Survey's Earth Resources Observation System (EROS) Data Center. We try to provide the coordination and the veneer of user services, but really the bulk of what we have is in these large federal agency data centers. Since we are not an agency and don't have a budget, all joint activities have a lead agency. That's been our habit for many years, and it's been, in our view, successful. In this way, a vested interest takes on a task and the rest of us help.

     The DMWG monthly meetings provide interagency and interprogram planning and coordination. We do a lot of information exchange as well. Things are moving very rapidly. It's a forum for everybody from the private sector to other government representatives to come talk to us about what they are doing and to hear what we are doing.

     The other thing that we really strongly believe is that review by the National Research Council (NRC) should be integrated right into the process. Often we don't wait until we complete a project to ask for a refereed type of review. We often ask the NRC to come in when we are still in the planning stages to provide us with some expert advice. We have found this to be extremely beneficial.

     As I said at the outset, the GCDIS is really the core. We develop activities there, and we replicate them to the other systems as they are needed. The mission of the GCDIS is to deliver relevant global change data and information to scientists, educators, legislators, and the public in the United States and abroad. It is probably the largest environmental data coordinating activity in the world. I can't think of another one that is as big, because it covers data and information that would be in a clearinghouse but also includes some of the process science information. We have, in the past, tried to segregate what's in and what's out, and we found that we couldn't for just climate change. If you add the human dimensions, it makes the activity even larger.

     We provide the primary public view of the U.S. Global Change Research Program. We have developed a system in which we have metadata available through the Global Change Master Directory, which is linked to directories in other agencies through a World Wide Web site, as is common now. However, there are really only 70 or so nodes in those agencies.

     The Global Change Information Research Office is currently operated by the Center for International Earth Science Information Network (CIESIN). CIESIN provides some of our user services and outreach.

Figure 22.1
	Image to be provided.

     Figure 22.1 gives a flavor of our projects. We have a piece in the center that changes periodically. The Web site also includes agency information, programs, and documents that you can link to. One thing that I want to highlight is the feature called "Ask Dr. Global Change." It has told us a lot about the people who are looking at our site and what sort of information they want.

     Basically, when you click on this you can ask questions and a human being will answer you. We thought that this might unmanageable, but it has not been. Sometimes we get questions such as, Can you help me with my homework questions? We tend to try to direct students to information services where they can find the answer for themselves, as opposed to simply giving them the answer. We also get questions such as the one from a regional planning commission, Global change seems like it is going to change our infrastructure needs. How do we find out about that? We now see in these questions a lot of interest that relates to things that I've seen in the EPA's public access arena, which is the "in my backyard interest." An individual's interests, especially those of the general public, tend to be very local.

     Not only are these questions driving us, they also drive the Global Change Research Program. There is a national assessment that is looking at more regional issues across the United States. I believe there are 19 regions and perhaps 8 sectors. That means that even the Global Change Research Program is going to need access to better information to answer questions that are progressively more local, and the global change data and information generated by the program is only the tip of the iceberg. We have to reach out to the rest of the data sources, which are held everywhere from private organizations to state and local governments.

     This drives the need to provide data and information exchange at the lowest possible cost, which runs us smack up against a couple of other issues that you have heard about already related to cost recovery and intellectual property rights. These international data access and database protection issues have been articulated by previous speakers referring to actions in the World Meteorological Organization and the World Intellectual Property Organization. There are also the current domestic database protection initiatives you heard about. This is a very serious problem, because it threatens the fundamental part of what we feel to be the service we provide, which is full and open information exchange.

     Another strategic issue, which was alluded to yesterday, is, How long can a principal investigator (PI) hold on to data? Can PIs keep them in their notebooks, in their file drawers, or in their computers forever and keep publishing papers or just let them sit there? We have provided to the agencies some suggested language to be modified as necessary (because one size will not fit all), which helps set limits on exclusive use. Basically, as a PI, if you are going to apply for to a grant--and EPA has adopted this requirement in its global change grants--then you have to say in your proposal how you are going to provide these data, how these data are going to be made available. Also you have to agree to this up front. If you want the money, you have to provide the data, and you do it when you publish or when the project ends, whichever occurs first.

     We think this is probably a reasonable way to get universities off the hook for the new Freedom of Information Act requirement that you heard about yesterday from Representative Holt.¹ If indeed they are providing their data and information to a federal agency, the federal agency is subject, and always has been, to the Freedom of Information Act. The direct burden on a university goes away.

     I also said we do planning. We think management by objective is a good idea. We have produced a number of written plans and are publishing a directory of newly available global data sets every year. These are available on the Web and in hard copy. We also have annual input into the President's budget document "Our Changing Planet." We are now in an operational mode.

     The NRC has produced some very important and useful reports. For example, the U.S. National Committee for CODATA produced the Bits of Power² report which has been a tremendously helpful document. This shows some of the relationship between the planning that we do and the advice that we have gotten from the Academy. There is a whole list of others.

Figure 22.2
	Image to be provided.

     Figure 22.2 gives you an idea of GCDIS access levels. Since part of this discussion is about the Government Performance and Results Act (GPRA), you should know what we did before GPRA ever came into being, it was actually part of our 1995 planning. In the figure, level 1 is the highest level. Given the technology now, maybe the first column needs some revisiting. T1 may not be the best target now, but at that point it was because we were operating with gopher technology at that time. We were wondering if the World Wide Web was really going to come in and save us as we thought it might. This is the kind of thing that you can do to set goals for yourself, realistic and objective goals, so that you can tell whether you have met them or not. You don't have to measure how many users showed up at your site and count beans that way. You can actually get to substance.

     We haven't stopped planning. The National Environmental Change Information System (NECIS) is something we are just beginning to take on. It shows how we go through a process. Right now we are at square one. The concept began as a box in "Our Changing Planet 2000," which was labeled "The Global Environmental Change Information Service." We changed the name slightly to make it more relevant to what we actually thought would happen.

     We think that NECIS will evolve over the next decade. We can't do this out of existing resources, so we will have a big burden trying to come up with some sort of new start. It's potentially a whole new way for us to work together in government--local government working with federal government, working with state and private partners. However, the paradigm does not exist--at the moment. We don't have an example of how to do this.

     This is really something quite new. It really has the purpose of serving a nonscientist user community. Scientists are not communicating with the public in an understandable way. We provide data, but not information. The NECIS concept is similar to a much-expanded Ask Dr. Global Change, only you come up with these larger issues. We are working with the National Assessment Working Group in developing the relevant questions to tackle first. The group held regional workshops in which people identified questions, and we're going to draw on the subjects raised in these workshops.

     We are planning this work in a "glass house," which is an important concept. Too often we come up with a conclusion, and that's all we put out. We now have to show the supporting evidence behind a conclusion. I think all of this work has to be available. The data behind the decision have to be available as well. I think we have the means to do it within the Web environment, and we have the responsibility. We're going to start with a case study in the Southeast, looking at water resources. We are also planning to get advice all along from the NRC.

     Now, according to GPRA, we should have performance goals, even though as an interagency working group, we don't have the reporting requirements that agencies have. The agencies in which all of this work is implemented ultimately will be responsible for reporting. We have planned four performance objectives for fiscal years 2000-2002, and they seem to be commonsense things.

     We have planned the National Environmental Change Information System. We also want to broaden the "full and open" data access policy. (These are challenging goals, by the way. Nobody has gone into this with any sort of feelings that it is going to be easy.) We also want to improve GCDIS service to users. That's our base. We need to make incremental improvements. Finally, we want to meet the data and information management needs of the program that funds us, which seems kind of commonsense, but in fact, management by objective is commonsense. A lot of what underlies GPRA requires you to plan, to write things down, and to manage by objective--to use common sense.

     Since the DMWG is really the synergy that comes from all of the agencies, it is useful to look at the General Accounting Office review of the GPRA plans for all the agencies,³ which reveals some common themes in the agencies' reporting for last year. To paraphrase, there need to be stronger links between goals and measures. There also is the need to link goals and budgets. Interagency coordination has to be increased as well. (We think this is a good thing, because it is why we are in business in the first place; we're an interagency coordinating body.) Finally, human, capital, and information resources have to be identified.

     The NRC Committee on Science, Engineering, and Public Policy also issued some remarks regarding performance.⁴ It pointed out that there are differences between applied and basic research, and we feel that we are on the applied end. We don't fall into the category of not really knowing where we are going with something. Again, agency coordination needs strengthening. So we are hearing the same thing from another body. The NRC also stated the need for planning and recognized that peer review is one of the best ways to evaluate success and to gauge stature. So I think, from what I have been able to determine from people who have looked at GPRA and looked at the way we do interagency planning and coordination that we are there. Well, this is the good news.

     The bad news is that there are external limiting factors, and these are showstoppers at times. If you look at the budgets for data centers, which represent the core of what we do and where we really implement things, funding levels are either flat or declining at the same time that requirements are rising. The other issue is that the proposed database protection legislation threatens to limit full and open data exchange. These factors together amount to a one-two punch in which it is very difficult to do much more than reactive management. Because of these factors, we are forced, year by year, to react to the environment that presents itself, rather than being good managers and implementing the intent of GPRA. We have to be very careful and account for these intervening actions.

     So this is the view from the trenches. Management by objective is a good thing. I've said this a couple of times. Planning, coordination, and review are essential to success. Everybody seems to agree with that.

     As a final note, we shouldn't be afraid to count intangibles or set the bar high. Often when you get something like GPRA applied to an agency, the response is to back off and only do things that are safe. That's not going to take us where we need to go. That's not how we get our goals accomplished. It, again, is reactive, as opposed to proactive. Despite all of this, external factors can overwhelm the best of intents. I think this is really the message, and that's the view from the trenches.

Notes

¹ See Chapter 14, "Using Scientific and Technical Data in the National Interest," in these proceedings.

² National Research Council. 1997. Bits of Power: Issues in Global Access to Scientific Data, National Academy Press, Washington, D.C.

³ See General Accounting Office, 1999. Managing for Results: Opportunities for Continued Improvements in Agencies Performance Plans, (GAO/GGD/AIMD-99-215), GAO, Washington, D.C.

⁴ National Research Council. 1999. Evaluating Federal Research Programs: Research and the Government Performance and Results Act, National Academy Press, Washington, D.C.