Program management. The charge to the Committee includes the provision of recommendations for improving the SBIR program (although not for assessing its continued existence). That charge requires review and assessment of how each agency SBIR program operates, and—where possible—the identification of best practices and possible improvements.
The objective of the study is not to consider if SBIR should exist or not—Congress has already decided affirmatively on this question. Rather, the NRC Committee conducting this study is charged with providing assessment-based findings of the benefits and costs of SBIR (described in the Objectives section above) to improve public understanding of the program, as well as recommendations to improve the program’s effectiveness. It is also important to note that, in accordance with the Memorandum of Understanding and the Congressional mandate, the study will not seek to compare the value of one area with other areas; this task is the prerogative of the Congress and the Administration acting through the agencies. Instead, the study is concerned with the effective review of each area.
The project is divided into two phases. Phase I has focused on data collection and the development of the methodology. Per the agreement with the agencies as outlined in the Memorandum of Understanding,3 this Methodology was submitted to an intensive Academy review process, involving 12 reviewers with recognized expertise in economics, statistics, program evaluation, survey methodology, innovation policy, federal R&D programs, and both large and small high-tech firms.
Extensive revisions and elaborations were required as a result of this review and they are now reflected in this document. The second phase of the study will now implement the research methodology developed in Phase I. Phase I included an initial symposium for the program as a whole, followed by a number of committee meetings and a series of workshops to address the specific features of each agency's program. This phase has focused on the development of survey instruments, case study templates, and related research to the extent possible. Additional details regarding the study methodologies to be used have been deferred in some cases because they cannot be defined precisely until some initial Phase II work has been completed. At the conclusion of Phase I, the overall methodology and evaluation tools will be submitted for review. In Phase II, research papers on general topics will be commissioned, and preliminary results of field research will be assessed and cross-checked. A symposium will be convened to discuss publicly the results of the research, and final reports will be prepared for each agency and for the program as a whole.
The purpose of this document is to describe the methodological approaches developed under Phase I of the study. They build from the precedents established in several key studies already undertaken to evaluate various aspects of the SBIR. These studies have been successful because they identified the need for utilizing not just a single methodological approach, but rather a broad spectrum of approaches, in order to evaluate the SBIR from a number of different perspectives and criteria.
This diversity and flexibility in methodological approach are particularly appropriate given the heterogeneity of goals and procedures across the five agencies involved in the evaluation. Consequently, this document suggests a broad framework for methodological approaches that can serve to guide the research team when evaluating each particular agency in terms of the four criteria stated above. Table 1 illustrates some key assessment parameters and related measures to be considered in this study.4