Annex I:
Tasks to Further Develop and Implement the Methodology

Summarized below are a series of tasks to be undertaken in assessing the SBIR program.

Task 1: Collect and interpret information on the mission of each agency’s SBIR program. Initial research suggests that the missions of the five agencies will differ in varying dimensions, and, further, that divisions or sub-groups within each agency have unique missions associated with the SBIR program. This first task of mission definition will be an essential for designing survey questions and case studies, and interpreting evaluation results. In conjunction with this task, the Committee will utilize:

  • printed information about the SBIR program, including program descriptions as well as intramural (e.g., agency) and extramural (e.g., academic, consulting, public agency) studies of the program and databases related to the program;

  • Face-to-face meetings with agency administrators and managers of the SBIR program to collect and clarify institutional information, with an eye to understanding the subtleties of the agencies’ SBIR program missions, and their modes of operation

  • Discussions and presentations from public symposia convened by the Committee. Draft descriptions of the SBIR program will be prepared for each of the five agencies, using a common template. Based on each draft report, a summary matrix will be developed presenting similarities and differences in the SBIR program among the agencies.

Task 2: Collect and interpret information fundamental to a review of (1) the value to the federal research agencies of SBIR-funded projects; and (2) the quality of research being conducted by small business.1

Information associated with “value to” the Federal agency from SBIR projects will be drawn primarily from internal agency sources. Individuals directly associated with SBIR are in the best position to assess the relative worth of the program. This will include managers at the agency and, in the case of DOD and NIH, at the sub-agency level. Non-SBIR officials with senior positions in the agency may also be interviewed for their view of the “importance” of SBIR to the agency. This information could be collected though face-to-face meetings and/or through survey instruments directed to appropriate individuals within each agency. If instruments are used, they will rely on a working definition of “value” developed with input from appropriate individuals in each agency.

Information on value, collected through face-to-face meetings, an internal survey, or both, will be summarized in a draft description of the SBIR program. (See Task 1.)2 “Value” may also be determined indirectly through other indicators – for example, the allocation of non-SBIR resources to support SBIR management functions may be a key indicator.

Information associated with the “quality of research” being conducted could be collected externally from non-agency sources using several methods. For the case of basic research, Arnold and Balázs (1998) argue that the “quality should be assessed in terms of its potential usefulness to others.3 Of course, SBIR is not related to basic research but rather early stage development, but the standard of usefulness may also be applicable.

Bibliometrics is a commonly used proxy to gauge the potential usefulness of research to others, and will be considered in this study. This method measures the number of peer-reviewed articles and citations to research articles. The relevant data can be gathered from external sources, such as the ISI Web of Science and similar resource bases.4 Data on the number of patents directly associated with the research applied for and awarded and the citations to such patents are a complementary indicator. External awards for the significance of the research, such as the IR100 Awards, constitute another important measure of research quality.

Sources internal to each organization conducting the research are also important to identify the scale and quality of research. Information on research activity can most effectively be collected through surveys sent to the funded

1  

The Committee interprets “review” to mean a summary of facts. A review should not contain conclusions or recommendations.

2  

Based on the study’s goal as expressed in Task 6, one implicit element of value relates to the ability of the agency to meet certain procurement needs through the SBIR program.

3  

See Arnold, E., and Balázs, K. “Methods in the Evaluation of Publicly Funded Basic Research,” OECD Report, March 1998. Implicit in the Arnold and Balázs argument is a linear view of the innovation process, although each segment in the linear progression need not be within the same organization.

4  

Overviews of citation and bibliometric analyses are in Melkers, J. “Bibliometrics as a Tool for Analysis of R&D Inputs,” in Evaluating R&D Impacts: Methods and Practices (edited by B. Bozeman and J. Melkers), Boston: Kluwer Academic Publishers, 1993. See also Narin, F. and Hamilton, K.S. “Bibliometric Performance Measures,” Scientometrics, Summer 1996.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 78
Annex I: Tasks to Further Develop and Implement the Methodology Summarized below are a series of tasks to be undertaken in assessing the SBIR program. Task 1: Collect and interpret information on the mission of each agency’s SBIR program. Initial research suggests that the missions of the five agencies will differ in varying dimensions, and, further, that divisions or sub-groups within each agency have unique missions associated with the SBIR program. This first task of mission definition will be an essential for designing survey questions and case studies, and interpreting evaluation results. In conjunction with this task, the Committee will utilize: • printed information about the SBIR program, including program descriptions as well as intramural (e.g., agency) and extramural (e.g., academic, consulting, public agency) studies of the program and databases related to the program; • Face-to-face meetings with agency administrators and managers of the SBIR program to collect and clarify institutional information, with an eye to understanding the subtleties of the agencies’ SBIR program missions, and their modes of operation • Discussions and presentations from public symposia convened by the Committee. Draft descriptions of the SBIR program will be prepared for each of the five agencies, using a common template. Based on each draft report, a summary matrix will be developed presenting similarities and differences in the SBIR program among the agencies. Task 2: Collect and interpret information fundamental to a review of (1) the value to the federal research agencies of 1 SBIR-funded projects; and (2) the quality of research being conducted by small business. Information associated with “value to” the Federal agency from SBIR projects will be drawn primarily from internal agency sources. Individuals directly associated with SBIR are in the best position to assess the relative worth of the program. This will include managers at the agency and, in the case of DOD and NIH, at the sub-agency level. Non- SBIR officials with senior positions in the agency may also be interviewed for their view of the “importance” of SBIR to the agency. This information could be collected though face-to-face meetings and/or through survey instruments directed to appropriate individuals within each agency. If instruments are used, they will rely on a working definition of “value” developed with input from appropriate individuals in each agency. Information on value, collected through face-to-face meetings, an internal survey, or both, will be summarized in a draft description of the SBIR program. (See Task 1.)2 “Value” may also be determined indirectly through other indicators – for example, the allocation of non-SBIR resources to support SBIR management functions may be a key indicator. Information associated with the “quality of research” being conducted could be collected externally from non-agency sources using several methods. For the case of basic research, Arnold and Balázs (1998) argue that the “quality should be assessed in terms of its potential usefulness to others.3 Of course, SBIR is not related to basic research but rather early stage development, but the standard of usefulness may also be applicable. Bibliometrics is a commonly used proxy to gauge the potential usefulness of research to others, and will be considered in this study. This method measures the number of peer-reviewed articles and citations to research articles. The relevant data can be gathered from external sources, such as the ISI Web of Science and similar resource bases.4 Data on the number of patents directly associated with the research applied for and awarded and the citations to such patents are a complementary indicator. External awards for the significance of the research, such as the IR100 Awards, constitute another important measure of research quality. Sources internal to each organization conducting the research are also important to identify the scale and quality of research. Information on research activity can most effectively be collected through surveys sent to the funded 1 The Committee interprets “review” to mean a summary of facts. A review should not contain conclusions or recommendations. 2 Based on the study’s goal as expressed in Task 6, one implicit element of value relates to the ability of the agency to meet certain procurement needs through the SBIR program. 3 See Arnold, E., and Balázs, K. “Methods in the Evaluation of Publicly Funded Basic Research,” OECD Report, March 1998. Implicit in the Arnold and Balázs argument is a linear view of the innovation process, although each segment in the linear progression need not be within the same organization. 4 Overviews of citation and bibliometric analyses are in Melkers, J. “Bibliometrics as a Tool for Analysis of R&D Inputs,” in Evaluating R&D Impacts: Methods and Practices (edited by B. Bozeman and J. Melkers), Boston: Kluwer Academic Publishers, 1993. See also Narin, F. and Hamilton, K.S. “Bibliometric Performance Measures,” Scientometrics, Summer 1996. 78

OCR for page 78
organizations/agencies, and probably to program managers within those organizations.5 Another measure of quality specific to SBIR research is the utility of outputs to the funding agency and/or to the market. Information about commercialization will also come from the surveys and case studies.6 7 Task 3: Collect information and evaluate, using traditional metrics, the economic benefits of the SBIR program. Griliches (1958) and Mansfield (1977) pioneered the application of fundamental economic insight to the measurement of private and social rates of return to innovative investments.8 Streams of investment costs generate innovations and associated streams of economic benefits over time. Once identified and measured, these streams of costs and benefits are used to calculate such performance metrics as social rates of return and benefit-to-cost ratios. Thus, the evaluation question that can be answered from this traditional approach is: Given the investment costs and the social benefits, what is the social rate of return from the innovation? The economic benefits achieved by the SBIR program can be evaluated using several methods, including survey and case study methods. Information collected in Task 1 and Task 2 will underpin the details of the approach. The evaluation literature and the evaluation experience of the Committee and that of the expert consultants reporting to the Committee,9 suggests that the first-level net benefits will be quantified based on both retrospective and prospective survey data. The information collected in Task 1 and Task 2 should identify relevant first-level output measures such as sales, employment growth, new products and processes, leveraged R&D investments (including additional R&D investment dollars as well as the establishment of new research partnerships10), and enhanced access to capital markets.11 The surveys will also include questions that address management issues. Second-level beneficiaries from the SBIR program include the agency that funded the project under evaluation. Third-level beneficiaries are the public- and private-sector consumers of the commercialized innovation developed by the award recipient. Both the evaluation literature and the evaluation experience of the Committee and others suggest that second- and third-level benefit data – quantitative and qualitative –can be collected through focused case studies. Task 3 relates to the second objective of this study. As noted above, part of the Congressional charge to the NRC is to compare the findings from Task 3 to evaluations of similar Federal research and development expenditures. Several Committee members and contract researchers have experience in evaluating Federal research and development programs. At the completion of Task 3, this expertise and experience will be applied to the task of assessing and evaluating the SBIR research results. 5 For an example of an analysis of NASA SBIR program managers’ qualitative information, see Archibald, R.B. and Finifter, D.H. “Evaluating the NASA Small Business Innovation Research Program: Preliminary Evidence of a Trade-off Between Commercialization and Basic Research,” Research Policy, April 2003. 6 Information about commercialization will also be collected from funded company officers and individual research scientists in a later task. 7 The Committee interprets “evaluation” to be a broader analysis than would be undertaken in an “impact assessment.” An impact assessment focuses on the impact (e.g., measured in terms of rates of return or benefit-to-cost comparisons) of the funded research on the agency’s stakeholders (e.g., small businesses). An evaluation includes an impact assessment as well as an examination of the portfolio of research vis-à-vis the objectives of the funding agency and an examination of how well the agency’s funding program are being managed. See Link, A.N. Economic Impact Assessment: Guidelines for Conducting and Interpreting Assessment Studies, Planning Report 96-1, National Institute of Standards and Technology, May 1996, for the application of these important terms as applied within the National Institute of Standards and Technology (NIST). See Georghiou, L., Dale, A., and Cameron, H. Special Issue of Research Evaluation on National Systems for Evaluation of R&D in the European Union, April for an application of these terms as applied within the European Union. As such, preliminary discussions with agencies suggest that a review of commercialization after the award would be useful to them for management purposes. The team anticipates viewing commercialization as an output of research and thus would logically become a part of the evaluation effort in this task. 8 See Griliches, Z. “Research Costs and Social Returns: Hybrid Corn and Related Innovations,” Journal of Political Economy, 1958. See also Mansfield, E., Rapoport, J., Romeo, A. Wagner, S., and Beardsley, G. “Social and Private Rates of Return from Industrial Innovations,” Quarterly Journal of Economics, 1977. 9 Some of the team members were involved in the evaluation of the Department of Defense’s Fast Track program. See National Research Council, SBIR: An Assessment of the Department of Defense Fast Track Initiative, 2000, op cit. 10 See Hagedoorn, J., Link, A.N., and Vonortas, N.S. “Research Partnerships,” Research Policy, April 2000. (2000) for a review of the theoretical and empirical literature related to research partnerships and R&D efficiency. 11 The Advanced Technology Program (ATP) within the National Institute of Standards and Technology (NIST) has a long and successful history of collecting through surveys such output measures to proxy first-level social benefits. See Ruegg, R.T. “The Advanced Technology Program, Its Evaluation Plan, and Progress in Implementation,” Journal of Technology Transfer, November 1997. See also Ruegg, R.T. and Feller, I. “A Toolkit for Evaluating Public R&D Investments: Models, Methods, and Findings from ATP’s First Decade,” NIST GCR 02-842, National Institute of Standards and Technology, May 2003. Finally, see the research papers contained in National Research Council, The Advanced Technology Program: Assessing Outcomes, C. Wessner (ed.). Washington, D.C.: National Academy Press, 2001. 79

OCR for page 78
Task 4: Collect and interpret information relevant to an evaluation of the non-economic benefits of the SBIR program. The Committee will explore how best to gauge the potential non-economic benefits of the SBIR program. Non economic benefits include the impact of SBIR on small business growth and development, knowledge effects, environmental benefits and public safety. These factors are related to cluster phenomena, links among SBIR firms, universities, government laboratories, and large firms, and the availability of highly qualified workers. Task 5: Collect and interpret information on Federal research and development funds to small businesses (between fiscal year 2000 and fiscal year 1983). Trend analysis is the appropriate methodology if “federal research and development funds to small businesses” is interpreted as support to small businesses through the SBIR program only. In such an analysis, two other factors must be controlled for: political factors associated with the supply of such funds and demand factors associated with changes in the extent of technological competition. A more comparative framework will be necessary if a broader definition, which includes other non-SBIR agency funding for small business R&D, is adopted. Task 6: Collect and interpret information on the extent to which SBIR Phase II awards fulfill the procurement needs of Federal agencies. Here, the Committee seeks to develop, particularly through case studies knowledge about how and why Federal agencies procure technology and how they use such technology. Implicit in Task 6 is the charge to understand the frontier associated with the effective use of SBIR Phase II technology, to understand how close Federal agencies are to that frontier, and to determine what factors are associated with such positioning. 80