Summary

Two surveys of the National Science Foundation’s Division of Science Resources Statistics (SRS)—the Survey of Federal Funds for Research and Development (the federal funds survey) and the Survey of Federal Science and Engineering Support to Universities, Colleges, and Non-profit Institutions (the federal support survey)—provide some of the most significant data available to understand research and development (R&D) spending and policy in the United States. Building blocks for virtually every analysis of U.S. scientific activity, they help reach conclusions about fundamental policy questions, such as whether a given field of research is being adequately funded, whether funding is balanced among fields, whether deficiencies in funding may be contributing to a loss of U.S. scientific or economic competitiveness, and which agencies are most important for the health of a scientific discipline. Budget officials at science agencies, Congress, and interest groups representing scientists, engineers, and high-technology industries, among others, constantly cite the survey results—or studies based on those results—in making public policy arguments.

However, the survey data are of insufficient quality and timeliness to support many of the demands put on them. For example, reporting agencies sometimes do not assign enough attention to proper recording and timely transmittal of the data. The surveys ask for information in categories that not all agencies use for their own internal purposes, so the information provided to SRS is often a rough estimate, frequently based on unexamined assumptions that originated years earlier.

Although the data from these two surveys have very important uses, the surveys are increasingly difficult to conduct in times of constrained



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 1
Summary T wo surveys of the National Science Foundation’s Division of Science Resources Statistics (SRS)—the Survey of Federal Funds for Research and Development (the federal funds survey) and the Survey of Fed- eral Science and Engineering Support to Universities, Colleges, and Non- profit Institutions (the federal support survey)—provide some of the most significant data available to understand research and development (R&D) spending and policy in the United States. Building blocks for virtually every analysis of U.S. scientific activity, they help reach conclusions about fundamental policy questions, such as whether a given field of research is being adequately funded, whether funding is balanced among fields, whether deficiencies in funding may be contributing to a loss of U.S. scien- tific or economic competitiveness, and which agencies are most important for the health of a scientific discipline. Budget officials at science agencies, Congress, and interest groups representing scientists, engineers, and high- technology industries, among others, constantly cite the survey results—or studies based on those results—in making public policy arguments. However, the survey data are of insufficient quality and timeliness to support many of the demands put on them. For example, reporting agencies sometimes do not assign enough attention to proper recording and timely transmittal of the data. The surveys ask for information in categories that not all agencies use for their own internal purposes, so the information provided to SRS is often a rough estimate, frequently based on unexamined assumptions that originated years earlier. Although the data from these two surveys have very important uses, the surveys are increasingly difficult to conduct in times of constrained 

OCR for page 1
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS resources, and their technological, procedural, and conceptual infrastruc- ture has not been modernized for procedure or content, in contrast with other surveys in the portfolio of SRS. SRS has recognized the need to upgrade these surveys and to implement recommendations from two previ- ous National Research Council (NRC) studies—Measuring the Science and Engineering Enterprise: Priorities for the Division of Science Resources Studies (2000) and Measuring Research and Development Expenditures in the U.S. Economy (2004)—which reviewed the federal funds and the federal support surveys as part of the broader SRS portfolio. With these issues in mind and at the request of the SRS, the Com- mittee on National Statistics of the National Research Council convened this panel to review the uses and collection of data on federal funds and federal support for science and technology and to recommend future directions for the program based on an assessment of these uses and the adequacy of the surveys. The panel was also asked to consider the classification structure, or taxonomy, for the fields of science and engi- neering, which provides the framework for the federal funds survey as well as other SRS surveys. The panel has engaged in a variety of activities as part of its respon- sibilities. We have reached out to senior officials of federal agencies that provide the federal funds data and key data users and solicited advice from providers of complementary and competing data sources. The panel also reviewed past studies on federal funds data, identified common require- ments, and considered new data elements and fields that could be useful to collect. As part of our information-gathering activities, the panel conducted a workshop on September 5-6, 2008, at which SRS and outside experts reviewed the uses and collection of data on federal funds for research and development, and assessed the adequacy of the surveys based on the uses. In the workshop, presenters addressed new and emerging methods of data access and retrieval, and recent federal government initiatives to increase the reliability and transparency of contract and grant databases. The workshop concluded with presentations on the issue of an appropriate fields of science classification structure. This report, with recommendations on modernizing the infrastructure of the survey, is the primary product of the study. The purpose of this report is to provide a pathway for SRS to follow, with the support of the Office of Management and Budget (OMB) and other federal agencies, in order to achieve some modest short-term improvements in the surveys while beginning to build a foundation for a much fuller, more useful R&D data system in the long term. In this report, we define the short term as the next 1 to 4 years; medium-term improvement actions are laid out for a period of 4 to 10 years; and long-term actions are understood to extend beyond the 10-year window. The timing of the pathway for change is outlined in the final chapter.

OCR for page 1
 SUMMARY In recognition of the constrained resources available for making changes in these surveys, the panel’s overarching conclusion is that it would be prudent for SRS to make a few short-term improvements to the current system of surveys and then to spend most of the available professional staff time and financial resources pursuing a solution in the medium and long term that involves making use of the new technology and automated databases that will soon be available. One issue to which resources should be devoted in the short run is to reconcile differences in the taxonomies used in the Survey of Federal Funds for Research and Development and its companion survey, the Survey of Research and Development Expenditures at Universities and Colleges (also known as the academic R&D expenditures survey). Recommendation 3-1: The Division of Science Resources Statistics, in the near term, should make the changes necessary to improve the comparability of the federal funds taxonomy and the taxonomy for the academic research and development expenditures survey and should focus on the medium- and long-term changes the panel recommends. The panel is convinced that high-level SRS staff involvement with responding agencies should go a long way toward demonstrating to them that SRS considers the data to be important and values their input. Much of the direct contact with reporting agencies has been relegated to the con- tractors who manage the data collection. The panel therefore recommends that SRS find the resources to establish formal linkages between its own staff and the individuals responsible for data collection and reporting in the various reporting agencies. Recommendation 3-2: The Division of Science Resources Statistics should devote staff and resources to managing relationships with responding agencies directly, relying less on contractors to maintain those relationships. The outreach effort would be assisted by the establishment of a more formal mechanism to achieve feedback on an ongoing basis and to pro- vide a forum for guidance as demonstration and evaluation projects are mounted. The panel notes that, in contrast to the companion federal sup- port survey and other SRS surveys that go out to the public, the federal funds survey does not provide respondents with any background on the law under which the data are collected, nor does it outline the important uses of the information. It is essential for SRS to regularly remind agencies about the authority for and importance of the survey. To accomplish this end, information about authority and uses could profitably be placed on the survey form and the associated website.

OCR for page 1
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS Recommendation 3-3: The Division of Science Resources Statistics should ensure that all questionnaires and email solicitations sent to respondents provide information on its data collection authority and on the important uses of the data. The timeliness of reporting is an issue affecting the quality of the R&D investment data. The data have diminished utility owing to the lags in their publication. One way of improving timeliness in this situation would be to impute or estimate the data for late respondents before all data have been received and to publish an estimate. However, some of the late reporters are among the largest supporters of R&D, and including estimates for their data in totals could lead to misleading results. It would be better to continue to work with the late reporters to improve the timeliness of their submissions. Recommendation 3-4: The Division of Science Resources Statistics (SRS) should maintain its current approach to data reporting, which is to wait for receipt of reports from all respondents before publishing the data. SRS should continue to report complete data without imputa- tion for missing reports and data elements. The agency should focus on working directly with respondents to find ways to improve the timeli- ness of their response to the surveys. Although SRS has developed web-based applications for reporting the data, the reporting formats do not make full use of the capabilities of the Internet, nor do they automate many of the functions that would make reporting easier for the responding agencies. Recommendation 3-5: The Division of Science Resources Statistics should invest in creating more user-friendly web surveys, possibly tai- lored to each agency, to replace current web versions of the paper surveys. Several government-wide initiatives hold promise of increasing the quality and availability of administrative records on government expen- ditures: the E-Government Act of 2002, the Federal Funding Accounting and Transparency Act of 2006 (Public Law 109-282), and recent initiatives on the part of OMB to put in place a significant program to standardize, enhance and validate the data that reside in the federal government’s con- tract and grant databases. These efforts also provide a means of meeting the need for data on R&D expenditures and other kinds of information that are now obtained by means of surveys and data calls. In addition, further improvements can be expected over time—for example, enhancing administrative records with identifiers or “tags” to assist in data retrieval will improve data access.

OCR for page 1
 SUMMARY Recommendation 4-1: The Division of Science Resources Statistics, in cooperation with the Office of Management and Budget and the Office of Science and Technology Policy, should seek to have all federal agen- cies that fund or conduct research and development (R&D) to incor- porate R&D descriptors (tags) into administrative databases. Ideally, in order to enable identification of the R&D components of agency or program budgets, tags should identify: the specific field of science and engineering; whether a record applies to R&D or R&D plant; and whether the record activity is basic research, applied research, or development. Other data enhancements are possible as the databases are improved in the long term. One necessary enhancement is to enrich the administrative databases with information on intramural R&D activities—those R&D activities conducted in agency-operated laboratories and other facilities that are not likely to be reflected in the contract and grant data systems. Recommendation 4-2: The Division of Science Resources Statistics should work with the Office of Management and Budget to seek endorsement to work with other research and development funding agencies to incorporate intramural data into existing and future data- bases or to directly access intramural spending information from per- former databases. Demonstration projects offer the opportunity for SRS to work with reporting agencies and other stakeholders to achieve their collective needs. Recommendation 4-3: The Division of Science Resources Statistics should initiate work with other federal agencies to develop several demonstration projects to test for the best methods to move to a system based at least partly on administrative records. The policy context will define the future demand for information on federal R&D expenditures. Novel approaches, such as data federation, automatic text, and linkage analysis, promise to contribute to the develop- ment of federal R&D databases rich with detail and increasingly transpar- ent and usable. The panel has identified some promising possibilities for the future of collecting information on federal R&D spending. This report lays out the pathway for SRS to follow in order to move the collection of data on federal R&D spending from today’s survey-centric model to an administrative data–based system. This could be accomplished in a series of overlapping steps, beginning with a series of modest improve- ments. At the same time as these modest improvements are under way,

OCR for page 1
 DATA ON FEDERAL RESEARCH AND DEVELOPMENT INVESTMENTS SRS should begin serious coordination with OMB, most likely through the Office of Science and Technology Policy, to initiate a process that will lead to additional data items being incorporated into the administrative databases. Taking into account lessons learned in the development of the National Institutes of Health’s comprehensive database system (the Research, Con- dition, and Disease Categorization system), SRS has the opportunity to promulgate similar comprehensive systems in other agencies—systems that incorporate taxonomic elements and permit cross-walks between programs, projects, and fields.