Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 1
--> Executive Summary The Government Performance and Results Act (GPRA), enacted in 1993, focuses agency and oversight attention on the performance and results of government activities by requiring that all federal agencies measure and report on the results of their activities annually. Agencies are required to develop a strategic plan that sets goals and objectives for at least a 5-year period, an annual performance plan that translates the goals of the strategic plan into annual targets, and an annual performance report that demonstrates whether the targets are met. The Committee on Science, Engineering, and Public Policy (COSEPUP) of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine has addressed the issue of measuring and evaluating research in compliance with the requirements of GPRA. COSEPUP recognizes the opportunities and challenges that GPRA presents for agencies that invest in research. GPRA offers those agencies the opportunity to communicate to policy-makers and the public the rationale for and results of their research programs. At the same time, GPRA presents substantial challenges to the agencies. During the course of this study, COSEPUP held several workshops. In these workshops and in other input to the committee, we have heard two distinct and conflicting viewpoints on approaches to measuring basic research. One is that it should be possible to measure research, including basic research, annually and provide quantitative measures of the useful outcomes of both basic and applied research. The other is that, given the long-range
OCR for page 2
--> nature of basic research, there is no sensible way to respond to the GPRA annual measurement requirement and that the best that can be done is to provide measures that appear to respond but in fact are essentially meaningless, such as a list of an agency's top 100 discoveries of the preceding year. COSEPUP's view, spelled out in more detail in what follows, is different from both those viewpoints. In essence, our report takes two strong positions. First, the useful outcomes of basic research cannot be measured directly on an annual basis, because the usefulness of new basic knowledge is inherently too unpredictable; so the usefulness of basic research must be measured by historical reviews based on a much longer timeframe. Second, that does not mean that there are no meaningful measures of performance of basic research while the research is in progress; in fact, the committee believes that there are meaningful measures of quality, relevance, and leadership that are good predictors of eventual usefulness, that these measures can be reported regularly, and that they represent a sound way to ensure that the country is getting a good return on its basic research investment. The problem of reporting on applied research is much simpler: it consists of systematically applying methods widely used in industry and in some parts of government. For example, an applied research program usually includes a series of milestones that should be achieved by particular times and a description of the intended final outcomes and their significance. Periodic reporting can indicate progress toward those milestones. The remainder of this executive summary provides a more in-depth description of COSEPUP's conclusions and recommendations regarding how to evaluate federal research programs relative to GPRA. It also addresses coordination among federal research programs and human-resource issues. COSEPUP concludes that both basic research and applied research programs1 can be meaningfully evaluated on a regular basis. For the applied research programs of the mission agencies, specific practical outcomes can be
OCR for page 3
--> documented and progress toward their achievement can be measured annually. For example, if the Department of Energy adopted the goal of producing cheaper solar energy, it could measure the results of research directed toward decreasing the cost of solar cells; this applied research project would be evaluated annually against specific measurable milestones. However, the practical outcomes of basic research in science and engineering can seldom be identified while the research is in progress. Basic research has annual results that can be meaningfully evaluated, but these evaluations often do not give even a hint of ultimate practical outcomes. History tells us unmistakably that by any measure, the benefit to the United States for leadership in basic research is extremely high—lives saved, inventions fostered, and jobs and wealth created. History also shows us how often basic research in science and engineering leads to outcomes that were unexpected or took many years or even decades to emerge. COSEPUP strongly believes that measures of the practical outcomes of basic research usually must be retrospective and historical and that the unpredictable nature of practical outcomes is an inherent and unalterable feature of basic research. For example, pre-World War II basic research on atomic structure contributed to today's Global Positioning System, an outcome of great practical and economic value, but, attempts to evaluate a year's worth of that early research even if they demonstrated high quality and world leadership, would have contained no hint of this particular outcome. Since we cannot predict the ultimate practical outcomes of basic research, we must find ways to ensure that the basic research programs that the nation funds generate the kinds of knowledge that have given us great practical benefits in the past. To do that, we must find ways to measure the quality of our current research programs, their contributions to our world leadership in the relevant fields, and their relevance to agency goals and intended users. World leadership is an important measure. In an earlier report (COSEPUP, 1993), COSEPUP recommended that, for the
OCR for page 4
--> sake of the nation's well-being, the United States be among the leaders in all major fields of science and pre-eminent in selected fields of national importance. That is because a nation must be performing research at the forefront of a field if it is to understand, appropriate, and capitalize on current advances in that field, no matter where in the world they occur. New knowledge has value to nations where highly educated people performing cutting-edge research in the field of discovery can make use of the new knowledge when practical outcomes appear possible. The people best qualified to evaluate basic or applied research are those with the knowledge and experience to understand its quality, and, in the case of applied research, its connection to public and agency goals. Evaluating basic research requires substantial scientific or engineering knowledge. Evaluating applied research requires, in addition, the ability to recognize its potential applicability to practical problems. With those considerations in mind, COSEPUP has reached six conclusions and offers six recommendations regarding the evaluation of federally supported research programs. Conclusion 1: Both applied research and basic research programs supported by the federal government can be evaluated meaningfully on a regular basis. Conclusion 2: Agencies must evaluate their research programs by using measurements that match the character of the research. Differences in the character of the research will lead to differences in the appropriate timescale for measurement, in what is measurable and what is not, and in the expertise needed by those who contribute to the measurement process.
OCR for page 5
--> For applied research programs, progress toward specified practical outcomes can usually be measured annually by using milestones and other fairly standard approaches common in industry and in some parts of the federal government. For basic research, in contrast, progress toward practical outcomes cannot be measured annually, and attempts to measure such progress annually can in fact be harmful. Basic research progress can be reported annually in terms of quality, leadership, and relevance to agency goals, but practical outcomes can be measured only against a far longer historical perspective. In practical terms, because quality, leadership, and relevance will usually change slowly, the GPRA annual-reporting requirement can usually be met by minor updating of full evaluations that are done in a more flexible timeframe. There is a much greater chance that important events will take place in subfields, because of either scientific events or funding changes, so subprogram changes should constitute much of the updating. Different expertise is required for measuring the worth of applied research and the worth of basic research. Measuring both requires technical and scientific knowledge, but applied research entails some factors that basic research does not, such as ultimate usability, so the input of potential users is required. That leads to our next conclusion. Conclusion 3: The most effective means of evaluating federally funded research programs is expert review. Expert review—which includes quality review, relevance review, and benchmarking—should be used to assess both basic research and applied research programs. Expert review is widely applied—used, for example, by congressional committees, by other professions, by industry boards, and throughout the realm of science and engineering—to answer complex questions through consultation with expert advisers. It is
OCR for page 6
--> useful in helping an agency answer three kinds of questions of particular relevance to GPRA: What is the quality of the research program—for example, how good is current research work compared with other work being conducted in the field?2 This question is best answered by reviewers who are sufficiently expert in the field being assessed to perform a quality review. This approach is traditionally called peer review. Peer review is commonly applied to projects, but here we are applying it to programs. The talent, objective judgment, and experience of these experts, or peers, are paramount and should be the criteria for their selection. Is the research program focused on the subjects most relevant to the agency mission? Another form of expert review is relevance review , in which potential users, joined by experts in related fields, evaluate the relevance of research to agency goals—is the research on subjects in which new understanding could be important in fulfilling the agency's mission? In reviewing the relevance of a program, a panel would assess the appropriateness of the direction of the research to the agency mission and its potential value to intended users. Is the research being performed at the forefront of scientific and technological knowledge? This is a relevant question for many programs, but it is particularly important for whole fields and subfields being supported. Evaluations of fields and subfields is best done through international benchmarking by a panel of experts who have sufficient stature and perspective to assess the international standing of research. For agencies whose missions include a specific responsibility for basic research—such as the National Science Foundation in broad fields of science and engineering, the National Institutes of Health in fields related to health, or the Department of Energy in high-energy physics—world leadership in a field can itself be an
OCR for page 7
--> agency goal. That is equally true for mission agencies, such as Department of Defense (DOD) but in more focused ways. For example, DOD can take as a goal world leadership in basic materials research relevant to its mission. Once such a goal is established, the usual measures of quality and leadership should be applied. Conclusion 4: The nation cannot benefit from advances in science and technology without a continuing supply of well-educated and well-trained scientists and engineers. Without such a flow, the capability of an agency to fulfill its mission will be compromised. Agencies must pay increased attention to their human-resource requirements in terms of training and educating young scientists and engineers and in terms of providing an adequate supply of scientists and engineers to academe, industry, and federal laboratories. Federal agencies that support research and exploit its results are able to do so because the education and training programs of the universities, in the course of performing much of that research, and the federal laboratories provide a continuing flow of qualified scientists and engineers. Even though section 1115(a)(3) of GPRA requires agencies to describe the human resources required to meet their performance goals, few agencies describe the importance of human resources or propose ways to ensure their adequacy in their strategic or performance plans. Conclusion 5: Mechanisms for coordinating research programs in multiple agencies whose fields or subject matters overlap are insufficient. It is common and valuable for agencies to approach similar fields of research from different perspectives. Indeed, this pluralism is a major strength of the U.S. research enterprise. But, better communication among agencies would enhance opportunities for
OCR for page 8
--> collaboration, help keep important questions from being overlooked, and reduce instances of inefficient duplication of effort. Present mechanisms need strengthening. Conclusion 6: The development of effective methods for evaluating and reporting performance requires the participation of the scientific and engineering community, whose members will necessarily be involved in expert review. The researchers who work in agency, university, and industrial laboratories are the people who perform and best understand the research programs funded by the federal government. Many researchers contribute substantial time and effort to reviewing papers submitted for publication, grant applications, and program proposals, yet few of them are aware of GPRA, its objectives, and its mandates. Increased contact with and advice from the broader scientific and engineering community regarding the methods of determining and reporting quality and regarding the leadership position of agency research programs and the relevance of research to agency missions can benefit the GPRA process. On the basis of those conclusions, COSEPUP offers the following recommendations: Recommendation 1: Because both applied research and basic research can be evaluated meaningfully on a regular basis and are vital ta research and mission agencies, research programs should be described in strategic and performance plans and evaluated in performance reports. The performance of research is critical to the missions of many federal agencies. Therefore, a full description of an agency's goals and results, which is a principal objective of GPRA, must contain an evaluation of research activities and their relevance to the agency's mission.
OCR for page 9
--> Recommendation 2: For applied research programs, agencies should measure progress toward practical outcomes. For basic research programs, agencies should measure quality, relevance, and leadership. In addition, agencies should conduct periodic reviews of the overall practical outcomes of an agency's overall past support of applied and basic research. The use of measurements needs to recognize what can and cannot be measured. Misuse of measurement can lead to strongly negative results; for example, measuring basic research on the basis of short-term relevance would be extremely destructive to quality work. Because the evaluation of applied research is directly connected to practical outcomes, whereas the evaluation of basic research is in terms of quality, relevance, and leadership, which ultimately lead to practical outcomes, there might be a tendency to bias an agency's overall research program toward applied research at the expense of basic research. This should be avoided, and a proper balance should be maintained. Recommendation 3: Federal agencies should use expert review to assess the quality of research they support, the relevance of that research to their mission, and the leadership of the research. Expert review must strive for balance between having the most knowledgeable and the most independent individuals serve as members. Each agency should develop clear, explicit guidance with regard to structuring and employing expert review processes. The most effective way to evaluate research programs is by expert review. The most commonly used form of expert review of quality is peer review. This operates on the premise that the
OCR for page 10
--> people best qualified to judge the quality of research are experts in the field of research. This premise prevails across the research spectrum, from basic research to applied research. A second form of expert review is relevance review, in which potential users and experts in other fields or disciplines related to an agency's mission or to the potential application of the research evaluate the relevance of research to the agency's mission. A third form of expert review is benchmarking, in which an international panel of experts compares the level of leadership of a research program relative to research being performed worldwide. Recommendation 4: Both research and mission agencies should describe in their strategic and performance plans the goal of developing and maintaining adequate human resources in fields critical to their missions both at the national level and in their agencies. Human resources should become a part of the evaluation of a research program along with the program's quality in terms of research advancement, relevance in terms of application development, and leadership in terms of the ability to take advantage of opportunities when they arise. In early drafts of strategic and performance plans, agencies have generally omitted discussions of education and training, which are fundamental to the ability of agencies to fulfill their missions. The goal of developing and maintaining adequate human resources in fields critical to their missions should be supported by plans that produce that outcome. The nation cannot benefit from advances in science and technology without a continuing supply of well-educated and well-trained scientists and engineers. In addition, in the absence of such a flow, the capability of an agency to fulfill its mission will be compromised and the knowledge learned and technology developed will be lost.
OCR for page 11
--> Recommendation 5: Although GPRA is conducted agency-by-agency, a formal process should be established to identify and coordinate areas of research that are supported by multiple agencies. A lead agency should be identified for each field of research and that agency should be responsible for assuring that coordination occurs among the agencies. It is common and valuable for multiple agencies to approach similar fields of research from different perspectives. Indeed, this pluralism is a major strength of the U.S. research enterprise. However, better communication among agencies would enhance opportunities for collaboration, help to keep important questions from being overlooked, and reduce instances of inefficient duplication of effort. A single agency should be identified to serve as the focal point for each particular field of research so that all significant supported fields are covered. Information regarding support for that field should be provided to all the agencies involved in it so that they can adjust their efforts to ensure that the field is appropriately covered. Agencies should use benchmarking, which affords the opportunity to look across fields, in their efforts to understand the status of a particular field of research. Recommendation 6: The science and engineering community can and should play an important role in GPRA implementation. As a first step, they should become familiar with agency strategic and performance plans, which are available on the agencies' web sites. The researchers who work in agency, university, and industrial laboratories are the people who perform and best understand the research programs funded by the federal government. Many researchers contribute substantial time and effort to reviewing papers submitted for publication, grant applications, and
OCR for page 12
--> program proposals, but few of them are aware of GPRA. Their greater involvement in implementing GPRA would be beneficial to the country. Increased contact with and advice from the broader scientific and engineering community regarding both the quality and the leadership position of agency research programs and the relevance of the research to agency missions can benefit the GPRA process. COSEPUP intends to address mechanisms and guidelines for implementing these recommendations in workshops and meetings with representatives from federal agencies, Congress, OMB, and oversight bodies. Given the diverse portfolio of research conducted by federal agencies and the urgency of addressing the question of how basic research can be evaluated in the context of GPRA, the level of detail and specificity needed in designing procedures and guidelines for implementation was beyond the scope of this report. The Government Performance and Results Act provides an opportunity for the research community to ensure the effective use of the nation's research resources in meeting national needs and to articulate to policy-makers and the public the rationale for and results of research. We believe that our recommendations can assist federal agencies in complying with GPRA. Notes 1. For purposes of this study, program refers to a set of activities focused on a particular area that can include multiple projects with different risks, time horizons, and outcomes. 2. There are at least two aspects of quality—one absolute and one relative. The absolute aspects are related to the quality of the research plan, the methods by which it is being pursued, its role in education when conducted at a university, and the importance of its results to its sponsor, either obtained or expected. The relative aspects pertain to its leadership at the edge of an advancing field. Although the leadership aspect is generally important, the results might in some cases be of great importance to an agency albeit not at the leading edge of a field.
Representative terms from entire chapter: