2
Survey Analysis1

As we have seen in the previous chapter, the Fast Track and Phase II Enhancement programs at the Department of Defense, while sharing the common objective of supporting SBIR commercialization, are different programs with largely different objectives. The analysis in this chapter thus focuses on two separate comparisons based on surveys of DoD SBIR awards: that between Fast Track award winners and a control group, and that between Phase II Enhancement award winners and a control group.2

2.1
SUMMARY OF SURVEY FINDINGS

The NRC survey generated a range of useful data related to characteristics and outcomes for FT and PIIE projects in relation to the control group:

  1. Demographics of each group.

  2. Outcomes from awards.

  3. Gap reduction.

1

The survey methodology for this update of the DoD Phase II Enhancement program is described in Appendix A.

2

This chapter presents an exposition and analysis of the survey responses for the three sample groups: Fast Track, Phase II Enhancement, and the population matched control group. The control group refers to subjects that did not participate in the Fast Track or Phase II Enhancement programs, but in all other respects were treated in the same way as the experimental group. Appendix B contains the questions on the Firm Survey and a summary of the individual responses. The 232 respondents answered 18 questions, many containing multiple parts; thus the complete data display is 232 by 72 fields and includes some answers (fields) that are at least one sentence long. Hence, summaries are the only reasonable way to publish the data. Appendix C contains the questions on the Project Survey and a summary of the individual responses. The 240 respondents answered 36 questions, many containing multiple parts. The NRC surveyed all companies receiving Phase II awards during the study period. Firms with multiple awards received more than one project survey. As a result, there are more project surveys than firm surveys.



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 28
2 Survey Analysis1 As we have seen in the previous chapter, the Fast Track and Phase II Enhancement programs at the Department of Defense, while sharing the common objective of supporting SBIR commercialization, are different programs with largely different objectives. The analysis in this chapter thus focuses on two separate comparisons based on surveys of DoD SBIR awards: that between Fast Track award winners and a control group, and that between Phase II Enhancement award winners and a control group.2 2.1 SUMMARY OF SURVEY FINDINGS The NRC survey generated a range of useful data related to characteristics and outcomes for FT and PIIE projects in relation to the control group: 1. Demographics of each group. 2. Outcomes from awards. 3. Gap reduction. 1 The survey methodology for this update of the DoD Phase II Enhancement program is described in Appendix A. 2 This chapter presents an exposition and analysis of the survey responses for the three sample groups: Fast Track, Phase II Enhancement, and the population matched control group. The control group refers to subjects that did not participate in the Fast Track or Phase II Enhancement programs, but in all other respects were treated in the same way as the experimental group. Appendix B contains the questions on the Firm Survey and a summary of the individual responses. The 232 respondents answered 18 questions, many containing multiple parts; thus the complete data display is 232 by 72 fields and includes some answers (fields) that are at least one sentence long. Hence, summaries are the only reasonable way to publish the data. Appendix C contains the questions on the Project Survey and a summary of the individual responses. The 240 respondents answered 36 questions, many containing multiple parts. The NRC surveyed all companies receiving Phase II awards during the study period. Firms with multiple awards received more than one project survey. As a result, there are more project surveys than firm surveys. 28

OCR for page 28
SURVEY ANALYSIS 29 2.1.1 Survey Demographics The survey aimed to reach projects that fell into three groups: • Fast Track (FT). The survey sought responses from the 217 FT awards made from 1997-2002. • Phase II Enhancement (PIIE) awards. The survey selected a sample of 210 PIIE awards. (The sample was weighted to adjust for the year of award). Thirty-four also received FT awards.3 • PIIE and FT. The survey also addressed 34 projects that received both FT and PIIE awards.4 • Control group (neither FT nor PIIE). The sample of 376 was weighted to ensure that the number of awards per year and the source of awards by DoD service was similar to those for the FT and PIIE projects samples combined.5 Thus, in total, 837 (217+210+34+376=837) projects were surveyed. Responses are provided in the Table 2-1. TABLE 2-1 Survey Response Rates by Category Award Surveys Email Could Not Completed Overall Contacted Award Contact Contact Survey Response Response (Number) (Number) (Number) Rate (%) Rate (%) Category Fast Track 156 61 50 23 32 Phase II 198 12 69 33 35 Enhancement Both Fast 32 2 14 41 44 Track and Phase II Enhancment Control 331 45 107 28 32 Group SOURCE: NRC Project Survey 2006. NOTE: Fourteen responses were received from projects that received both FT and PIIE awards. In aggregating the data, responses from these projects were added to both the FT and PIIE categories. 3 See Appendix A, footnote 5, for details. 4 For a description of the sample selection, see Appendix A. 5 The control group sample was larger to account for the possibility that these projects might generate a lower response rate.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 30 BOX 2-1 Multiple Sources of Bias in Survey Response Large innovation surveys involve multiple sources of bias that can skew the results in both directions. Some common survey biases are noted below.a Successful and more recently funded firms are more likely to respond. o Research by Link and Scott demonstrates that the probability of obtaining research project information by survey decreases for less recently funded projects and it increased the greater the award amount.b Nearly 40 percent of respondents in the NRC Project Survey began Phase I efforts after 1998, partly because the number of Phase I awards increased, starting in the mid 1990s, and partly because winners from more distant years are harder to reach. They are harder to reach as time goes on because small businesses regularly cease operations, are acquired, merge, or lose staff with knowledge of SBIR awards. Success is self-reported. Self-reporting can be a source of bias, although o the dimensions and direction of that bias are not necessarily clear. In any case, policy analysis has a long history of relying on self-reported performance measures to represent market-based performance measures. Participants in such retrospectively analyses are believed to be able to consider a broader set of allocation options, thus making the evaluation more realistic than data based on third party observation.c In short, company founders and/or principal investigators are in many cases simply the best source of information available. Survey sampled projects at firms with multiple awards. Projects from o firms with multiple awards were under-represented in the sample, because they could not be expected to complete a questionnaire for each of dozens or even hundreds of awards. Failed firms are difficult to contact. Survey experts point to an o “asymmetry” in their ability to include failed firms for follow-up surveys in cases where the firms no longer exist.d It is worth noting that one cannot necessarily infer that the SBIR project failed; what is known is only that the firm no longer exists. Not all successful projects are captured. For similar reasons, the NRC o Project Survey could not include ongoing results from successful projects in firms that merged or were acquired before and/or after commercialization of the project’s technology. The survey also did not capture projects of firms that did not respond to the NRC invitation to participate in the assessment. Some firms may not want to fully acknowledge SBIR contribution to o project success. Some firms may be unwilling to acknowledge that they received important benefits from participating in public programs for a

OCR for page 28
SURVEY ANALYSIS 31 variety of reasons. For example, some may understandably attribute success exclusively to their own efforts. Commercialization lag. While the NRC Project Survey broke new ground o in data collection, the amount of sales made—and indeed the number of projects that generate sales—are inevitably undercounted in a snapshot survey taken at a single point in time. Based on successive data sets collected from NIH SBIR award recipients, it is estimated that total sales from all responding projects will likely be on the order of 50 percent greater than can be captured in a single survey.e This underscores the importance of follow-on research based on the now-established survey methodology. 14 12 Sales (Millions of Dollars) 10 8 6 4 2 0 1 2 3 4 5 6 7 8 9 10 11 12 Survey Taken Years after Phase II Award FIGURE B-2-1 Survey bias due to commercialization lag. Figure B-2-1 These sources of bias provide a context for understanding the response rates to the NRC Survey conducted for this study. continued

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 32 Limited statistical analysis. Owing to the constraints imposed in some o cases by low sample size, more sophisticated statistical analysis could not be performed. Accordingly, while the analysis below compares a control group of projects with Fast Track and Phase IIE projects respectively, different outcomes may not always be statistically significant. a For a technical explanation of the sample approaches and issues related to the NRC surveys, see Appendix A. b Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, London: Routledge, 2005. c While economic theory is formulated on what is called “revealed preferences,” meaning individuals and firms reveal how they value scarce resources by how they allocate those resources within a market framework, quite often expressed preferences are a better source of information especially from an evaluation perspective. Strict adherence to a revealed preference paradigm could lead to misguided policy conclusions because the paradigm assumes that all policy choices are known and understood at the time that an individual or firm reveals its preferences and that all relevant markets for such preferences are operational. See {1} Gregory G. Dess and Donald W. Beard, "Dimensions of Organizational Task Environments," Administrative Science Quarterly, 29:52-73, 1984; {2} Albert N. Link and John T. Scott, Public Accountability: Evaluating Technology-Based Institutions, Norwell, MA: Kluwer Academic Publishers, 1998. d Albert N. Link, and John T. Scott, Evaluating Public Research Institutions: The U.S. Advanced Technology Program’s Intramural Research Initiative, op. cit. e Data from NIH indicates that a subsequent survey taken two years later would reveal very substantial increases in both the percentage of firms reaching the market, and in the amount of sales per project. See National Research Council, An Assessment of the SBIR Program at the National Institutes of Health, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2008. 2.1.2 Firm and Project Characteristics6 Phase II Enhancement (PIIE) and Fast Track (FT) are aimed at different objectives and have different requirements, so it is not surprising that the characteristics of firms in these programs were also different. Fast Track Projects Firms that tended to be smaller and founded more recently than the control group and also to have less SBIR experience undertook these projects. • Employees. Forty-three percent of Fast Track respondents reported having five employees or fewer at the time of the award, about a quarter more than the 36 percent in this category for the control group.7 6 All data provided in this report refer to NRC Fast Track and Phase II Recipient survey respondents in a particular category, not all awardees, unless specifically noted otherwise. 7 See Figure 2-2, NRC Project Survey 2006, Question 16, Appendix C.

OCR for page 28
SURVEY ANALYSIS 33 • Foundation year. The median year of foundation was 1995 for Fast Track recipients and 1988 for the control group. More than 2/3rds of the Fast Track firms were founded after 1992, compared with less than a third of the control group.8 • Prior Phase II awards. Almost half (48 percent) of FT winners reported zero prior Phase II awards, compared to 23 percent of control group firms.9 • Size of awards. FT projects received about $100,000 more in SBIR funding than did the control groups projects.10 Phase II Enhancement Projects Overall, firms with PIIE were more closely aligned with the experience profile of the control group: • Employees. PIIE firms reported that only 24 percent of firms had five or fewer employees. This is one third less than the control group.11 • Foundation year. PIIE firms were slightly younger than the control group, with a median foundation year of 1990 (compared with 1998).12 • Prior SBIR experience. PIIE and control group firms had almost identical prior SBIR experience, with about 24 percent having received zero prior SBIR Phase II awards, and 17-20 percent having received more than five prior Phase II awards.13 • Size of awards. As one important feature of PIIE is the provision of up to $500,000 in matching funds from SBIR, it is not surprising that on average PIIE projects received about $400,000 more than control group projects.14 Thus, the data indicate that younger firms with fewer employees have tended to utilize Fast Track, while PIIE firms are quite similar in profile to control group. Both received more funding than the control group, PIIE firms considerably more. 8 See Table App-A-3 in Appendix A 9 See Figure 2-4, NRC Project Survey 2006, Question 19, Appendix C. 10 See DoD Awards Database. 11 See Figure 2-5, NRC Project Survey 2006, Question 16b, Appendix C. 12 NRC Firm Survey 2006. Part A, Question 7, of the Firm survey asked firms the year their company was founded. 13 See NRC Project Survey 2006, Question 19. See Appendix C. 14 See DoD Awards Database.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 34 2.1.3 Project Outcomes The NRC methodology identified a number of metrics that can be used to develop an assessment of project outcomes for SBIR. These include: • Employment effects. • Sales revenues. • Sales by sector. • Additional investments. • The project initiation decision. • Project delays absent SBIR. • Knowledge effects. 2.1.3.1 Employment Responding firms from all three groups showed substantial employment gain between the time of the award and the time of the survey. • FT firms started from a much lower employment profile. They grew larger firms at about the same rate as the control group, but also rapidly reduced the share of firms with 1-5 employees, from 43 percent to 16 percent.15 • PIIE. Starting from about the same base, PIIE firms grew about twice as fast as the control group in the largest employment category (firms with 50 or more employees). Forty percent of responding PIIE firms were in this category at the time of the survey.16 2.1.3.2 Revenues The survey asked firms to attribute sales revenues (as well as expected sales and sales by licensees) to the related SBIR project. Given the high degree of skew in reported sales, where a few projects report very large positive results, it is important to ascertain both the overall level of projects that reach the market, and also their distribution. Reaching the market. All three groups reached the market at approximately the same rate17, which suggests that ceteris paribus, neither FT nor PIIE had a significant effect in enhancing the sales rate experienced by the control group’s projects (sales rate is defined as the percentage of projects that sold at least $1 in the marketplace).18 15 See Figure 2-5, NRC Project Survey 2006, Question 16b. See Appendix C. 16 Ibid. 17 See DoD Awards Database. 18 Use of the sales rate is valid only as one benchmark among many to capture the numerous dimensions of commercialization. However, it is still an important metric.

OCR for page 28
SURVEY ANALYSIS 35 Expected sales. FT and control group projects surveyed each reported that about 55 percent of those without sales to date still anticipated reaching the market at some point. About 66 percent of PIIE projects reported similar expectations.19 However, a comparison of expected sales from the previous (2000) survey and actual sales later reported indicates that firm descriptions of expected sales were highly optimistic, which suggests that expected sales reported in the current survey should be treated with caution. Distribution of sales by size. Given the high degree on skew in SBIR outcomes, in which a few projects account for a substantial share of total revenues generated by SBIR projects, it is important to review the distribution of sales by size. • FT. The distribution of sales for the Fast Track and Control Groups are quite similar. FT did not report a project with sales greater than $10 million, but it did report more projects with sales between $5 million and $10 million. 20 • PIIE. The PIIE group generated two awards with more than $10 million in cumulative revenues, and six with revenues of $5 million to $10 million, compared to one and two respectively for the control group.21 Given the very small numbers of awards involved, caution should be employed when drawing conclusions from these data. They are however suggestive. Sales by sector. One of the primary missions of the DoD SBIR program is to provide technology for use within DoD. Consequently, sales to DoD itself and to prime contractors for DoD are an important metric that the program is meeting its objectives. The survey reports substantial differences between the groups. In both cases, FT and PIIE projects are more likely to generate sales to DoD and its prime contractors than are the control group firms. • FT. About 70 percent of surveyed FT project sales by value were made to DoD and its primers. In contrast, this sector generated about 38 percent of sales for the control group. About 1/3rd of total revenues for FT projects came from direct sales to DoD, compared with 25 percent for the control group.22 • PIIE. Similarly, 65 percent of PIIE project revenues were from DoD and the primes. Direct sales to DoD in this case accounted for about 45 percent of the total.23 19 See NRC Project Survey 2006, Question 9. See Appendix C. 20 Ibid. 21 Ibid. 22 See NRC Project Survey 2006, Question 4. See Appendix C. 23 Ibid.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 36 2.1.3.3 Additional Investments Many projects require additional investment after Phase II before they can reach the market. These investments are therefore an additional metric that progress is being made with the project. Both the FT and PIIE programs require additional third party investment as a condition of the award, although the precise requirements are different. Consequently, both FT and PIIE projects generated much higher rates of additional investment than the control group. In this case then, the scale of additional investment may be significant. • FT. On average, FT projects attracted about $300,000 more in development funding than the control group. Interestingly, FT projects attracted smaller amounts of federal development funding, and considerably more private sector funding.24 This likely reflects the fact that FT projects must attract third party funding very early in the project cycle, when federal technology development funding is less available. Almost 35 percent of FT projects reported some kind of negotiations for the sale of equity, compared with 4 percent of the control group. And 14 percent had completed agreements at the time of the survey, compared with 1 percent of the control group.25 • PIIE. On average, PIIE projects generated just under $1 million in additional non-SBIR investment, in comparison with about $500,000 for the control group.26 That difference is entirely accounted for by the relative success of PIIE firms in attracting non-SBIR federal R&D funding (averaging just over $500,000 per project). 2.1.3.4 The Project Initiation Decision Firms were asked whether they would have undertaken the project absent the SBIR award in question, and also whether the SBIR. Data from the survey indicates that neither FT nor PIIE had a significant effect on this decision. • FT. Firms report that about the 66 percent of FT projects (66 percent) would certainly or probably not have been started without SBIR funding. This is approximately the same percentage as for the control group (70 percent).27 24 See NRC Project Survey 2006, Question 23. See Appendix C 25 See NRC Project Survey 2006, Question 12. See Appendix C. 26 See NRC Project Survey 2006, Question 23. See Appendix C. 27 See NRC Project Survey 2006, Question 13. See Appendix C.

OCR for page 28
SURVEY ANALYSIS 37 • PIIE. Here 75 percent of firms reported that the certainly or probably would not have started the project without SBIR—again, similar to results for the control group (70 percent).28 2.1.3.5 Project Delays Firms that believed their project would have proceeded without SBIR funding were further surveyed to determine whether those projects would have been significantly delayed without SBIR. • FT respondents indicated that on average their projects would have been delayed by 13 months, in comparison to 6 months for the control group.29 This is a substantial difference, which suggests that FT participants saw a particular value in the improved PI-PII transition offered via the FT program. • PIIE projects would have been delayed by 5 months, which is less than the control group.30 This relatively low anticipated delay may be based on the availability of the funding that was eventually used to make the PIIE match, which might have allowed rapid forward movement on the project even without SBIR. 2.1.4 Phase I-Phase II Funding Gap and Fast Track FT was designed explicitly to help reduce the funding gap that can occur between the end of Phase I funding and the beginning of Phase II. Survey data indicates that for responding projects, FT had a substantial impact in reducing funding gaps relative to those experienced by the control group. • Likelihood of funding gap. Survey data indicate that only about 35 percent of FT projects experienced a funding gap between PI and PII, in comparison with 70 percent of control group projects.31 • Length of funding gap. The average PI-PII funding gap for all FT projects was just over 1 month, compared with just under 6 months for the control group. For projects with some gap, FT projects averaged a gap of 4 months compared with more than 8 months for the control group.32 28 Ibid. 29 See NRC Project Survey 2006, Question 15a. See Appendix C. 30 Ibid. 31 See NRC Project Survey 2006, Question 31. See Appendix C. 32 Ibid.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 38 • Impact of the gap. Just under 20 percent of FT respondents reported that the funding gap had caused them to stop work on the project, compared to 40 percent of the control group projects.33 • Trends. Data from DoD indicates that the size of the gap has increased and the percentage of projects reporting a gap has increased in recent years.34 2.2 DETAILED SURVEY RESULTS In order to ensure that appropriate comparisons are made, this section is divided into two: the first part addresses comparisons between the Fast Track projects and the control group; the second compares responses from the Phase II Enhancement group and the control group. 2.2.1 Phase II Completion Respondents were asked to categorize the status of the efforts resulting from the surveyed Phase II awards. Some had already discontinued the effort, but most had realized some level of commercialization or were in post-Phase II (development of the technology) phase. Although all Phase II awards in the three sample groups35 had been made by the end of 2002, by the summer of 2006, 15 of the 240 responses had not yet completed Phase II. Two of these 15 had garnered both a Fast Track award and a Phase II Enhancement. The status chart (Figure 2-1) shows that firms benefiting from the Phase II Enhancement awards appeared more likely than the control group to have products, services, or processes in use by the target population.36 Fast Track projects were not more likely to be in the comem4rcialization stage than the control group. 33 See NRC Project Survey 2006, Question 33. See Appendix C. 34 Information from the DoD Fast Track Database. 35 Fast Track awards that subsequently received a Phase II Enhancement were analyzed in both groups; thus, Fast Track analyzed 64 responses and Phase II Enhancement, 83 responses. 36 The chart shows the percentage of each response from each sample. The low number of responses and the relatively small differences in responses for each sample does not support strong conclusions for this question

OCR for page 28
SURVEY ANALYSIS 63 Phase II Enhancement As Phase II Enhancement can obtain up to a half million dollars in additional SBIR money by obtaining appropriate matching funds, Figure 2-16 adds the SBIR funding to the funding shown in Figure 2-15. The average SBIR funding for the responding awards was $500,000 higher for Phase II Enhancement awards than for Control Group. Returning to the comparison of additional funding between the Phase II Enhancement and the control group on Figure 2-15, we can examine the totals without the input of Phase II Enhancement by third parties. As discussed above, Phase II Enhancement awards received, on average, $400,000 more in awards than the control group. Since Phase II Enhancement provides a dollar for dollar match of SBIR funding to third-party funding, it can be presumed that at least $400,000 of the additional average funding for Phase II Enhancement (displayed in Figure 2-15) was the funding that was executed during the additional year of Phase II. This $400,000 is less than 40 percent of the difference in average additional funding of Phase II Enhancement compared to the control group.86 2.2.6.2 Venture Capital Impacts Venture capital investment is often suggested as an alternative to SBIR.87 However, less than two percent of the awards reported any venture funding. Four Fast Track, one Control Group, and three Phase II Enhancement awards reported venture funding.88 As can be seen from Figure 2-15, the impact of venture capital on available funding was marginal. The average venture capital investment on these few projects was less than $1,000,000. The number of projects reporting VC funding was too small to allow substantive comparisons between groups. Enhancement, thus adding to their SBIR funding. When these 14 are removed from the Fast Track averages, the average SBIR for the remaining 50 Fast Track awards ($797,267) is comparable to that of the control group. 86 A similar direct comparison for Fast Track is not simple. Fast Track awards receive no additional SBIR funding compared to control group. The required third-party funding may be a dollar for dollar match to the Phase II award, or a one dollar to four SBIR dollars match dependent on whether this is the firm’s first SBIR Phase II. Since almost two-thirds of Fast Track respondents said this was their first Phase II, the average third-party match would be 1/3 x $800k +2/3 x $200k = $400k; however, almost one fourth of the Fast Track also received Phase II Enhancements, raising this average for third-party funding to about one half million dollars. Continuing this logic, one fifth of the Phase II Enhancement respondents were also Fast Track awards, hence the full average third- party funding for Phase II Enhancement awards was also almost a half million dollars. 87 The limitation on majority VC owned firms was implemented only beginning in 2002. For a review of the impact of the SBA rule on the SBIR program at NIH, see, National Research Council, Venture Capital Funding and the NIH SBIR Program, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009. 88 One of the awards reporting venture capital was both a Fast Track and a Phase II Enhancement.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 64 2.2.7 Knowledge Effects: Patents and Scientific Publications The congressional objectives for the SBIR program include additions to the nation’s scientific and technical knowledge. Patents, copyrights, and scientific publications are all evidence that this objective is being addressed by the SBIR program. The number of patents and copyrights applied for and issued is a measure of the intellectual property being generated. Since most scientific journals are refereed, the number of publications submitted and published measures to some degree the scientific merit of the SBIR. The numbers of patents, copyrights, trademarks, and scientific publications to date were measured by the survey.89 (See Figure 2-20.) Fast Track A concern expressed by some opponents of Fast Track is that, in order to attract third-party investors, there can be no true research in a Fast Track SBIR. The project must be much further down the development path. They contend that to obtain third-party financing, innovation must be complete or nearly complete and the SBIR merely serves to validate. If this is the case, one might expect a lower level of patent and copyright activity for Fast Track. 1.80 Average Number per Phase II Award 1.60 Fast Track 1.40 Phase II Enhancement 1.20 Control Group 1.00 0.80 0.60 0.40 0.20 0.00 Patents Copyrights Trademarks Publications Received Received Received Published FIGURE 2-20 Disclosure activity: Average number of patents, copyrights, Figure 2-20.eps trademarks, and scientific publications received for the technology developed as a result of project. SOURCE: NRC Project Survey 2006, Question 18. See Appendix C. 89 NRC Project Survey 2006, Question 18. See Appendix C.

OCR for page 28
SURVEY ANALYSIS 65 The data reported in the figure above indicate that Fast Track and Control Group projects tend to patent at much the same rate. Both Fast Track and Control Group projects reported 0.42 patents per Phase II award. Eighty six percent of FT projects reported zero patents, compared with 77 percent of Control Group projects.90 Phase II Enhancement The average patents reported received per phase II award varies from 0.30 for Phase II Enhancement to 0.42 for the control group.91 Once again, only a few firms reported receiving patents: 13 percent of PIIE firms and 23 percent of the Control Group.92 It therefore appears that patenting is a somewhat less important tool for protecting intellectual property among PIIE firms. It is also worth noting that, as in other area, skew is important: one firm accounted for a fifth of all Control group patents (9). 2.2.7.1 Scientific Publications The largest disclosure of research results appears to be in scientific publications. Three hundred and sixty-seven of the 380 papers reported as submitted to scientific publications were published. The control group included one award that reported 50 publications and another that reported 12 publications. Five awards (three Phase II Enhancement and one each Fast Track and Control Group) reported ten publications apiece. Most awards (74 of 107) that reported publication, had three or fewer papers published in scientific journals. Overall, there was little difference in the reported intellectual property/disclosure results for the vast majority of awards in each group. Slight differences in averages were due to single high performers in each category. 2.2.8 Impact of the Use of Fast Track and Phase II Enhancement on Responding Awards Respondents were asked several questions that were answered only by Fast Track awards or only by Phase II Enhancement Awards. The first question dealt with the source of matching funds in the Fast Track proposal or in the Phase II Enhancement proposal. (See Figure 2-21.) Since respondents could identify more than one source, the percentage totals of responses for the sources in Figure 2-18 add up to more than 100 percent. The number of respondents indicating their own firm as the source of matching funds in the proposal is quite unusual. Nothing prevents a firm from investing in its own SBIR research; however, DoD criteria for Fast Track 90 N=56 and n=107 respectively. 91 One control group Phase II award received 11 patents, which accounts for its advantage in averages. Ibid. 92 N=77 and n=107 respectively.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 66 100 Own firm 90 Federal agency Other firm 80 Percent of Respondents Angel investor or other 70 private funding source 60 Venture capital 50 Other (mostly state) 40 30 20 10 0 Fast Track Phase II Enhancement FIGURE 2-21 Proposal matching funds: Sources of matching or co-investment Figure 2-21.eps funding included in the proposal. SOURCE: NRC Project Survey 2006, Question 25. See Appendix C. specifically require that the matching funds in the proposal have to come from an outside third-party source. Fast Track The requirement to find third-party financing for Phase II and to obtain finalized agreements within 150 days of the award of Phase I has been a deterrent to participation in Fast Track. One-fourth of the Fast Track respondents reported that it took six months or more to obtain an agreement for third-party financing, indicating that they had to begin the search for financing prior to obtaining the Phase I award. The reported average time to obtain third- party financing for Fast Track was 3.5 months. The longest reported time was 12 months. Phase II Enhancement For Phase II Enhancement, the individual component programs have differing criteria for outside investors, some specifying only acquisition programs qualify and others allowing private sector. The tilt toward acquisition programs for Phase II Enhancement can be seen in the high percentage of federal agency shown as the source of matching funds for that program.

OCR for page 28
SURVEY ANALYSIS 67 For Phase II Enhancement, the average time to find and obtain an agreement for third-party financing was 4.1 months.93 Eleven percent of the respondents reported times of 12 or more months. Phase II Enhancement proposals are submitted on the schedule required by the individual components, but never prior to the completion of the first year of Phase II. Hence there is much more time available to locate and negotiate with the third-party source.94 The survey asked for comparison of the performance of Phase II Fast Track Awards and awards that also received Phase II Enhancement to performance of a standard Phase II award. As shown in Figure 2-22, the surveyed awards were judged to perform better, worse, or the same as a standard Phase II. Note that the relevant comparison in each case is between the control group and either Fast Track or PIIE. There is no relevant comparison between the latter groups. Note also that in the case of each variable, both Fast Track and PIIE were reported to compare favorably with the control group. 2.2.8.1 Speed to Market Over 50 percent of Fast Track and Phase II Enhancement respondents contended that the initial products reached the market faster than products of the average standard Phase II award. For Phase II Enhancement, which includes an extra year of Phase II development, seven percent stated that the time for the initial product to reach the market was longer than would have occurred absent the Phase II Enhancement.95 2.2.8.2 Sales to Date A substantial percentage of Fast Track (38 percent) and Phase II Enhancement (48 percent) projects claimed that sales to date were greater than would have occurred with a standard Phase II award. Similarly, both Fast Track (52 percent) and Phase II Enhancement (47 percent) projects claimed that investment to date was greater than would have occurred with a standard Phase II award. Seven percent (four awards) reported smaller investment to date than would have occurred absent the Phase II Enhancement.96 2.2.8.3 Potential Sales Potential sales were expected to be greater for Fast Track (45 percent) and for Phase II Enhancement (69 percent) projects than these groups would have expected to occur with a standard Phase II award. Investment in Phase II Enhancement may ease the transition into a DoD procurement. Seventy-five percent of the awards that reported such investment also reported improved transition. Similarly, 72 percent of the Fast Track projects that reported 93 NRC Project Survey 2006, Question 26. See Appendix C. 94 Ibid. 95 NRC Project Survey 2006, Question 27 and 29. See Appendix C. 96 See NRC Project Survey 2006, Question 27 and 29. See Appendix C.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 68 Worse Same Improved 100 Percent of Respondents 90 80 70 60 50 40 30 20 10 0 Enhancement Enhancement Enhancement Enhancement Enhancement Phase II Phase II Phase II Phase II Phase II Fast Track Fast Track Fast Track Fast Track Fast Track Time to Sales of Investment Potential Transition Reach Market Product Sales into DoD Procurement FIGURE 2-22 Comparisons to standard SBIR awards: Impact Fast Track or Figure 2-22.eps Phase II Enhancement had as compared to a standard Phase II proposal. SOURCE: NRC Project Survey 2006, Questions 27 and 29. See Appendix C. investment by a federal agency as their source of matching funds also reported improved transition.97 2.2.8.4 Satisfaction with Fast Track and the Phase II Enhancement Figure 2-23 shows the bottom line satisfaction by the firms with both the Fast Track program and the Phase II Enhancement program. Respondents were asked “In retrospect, knowing the outcome, are you satisfied with your decision to use” the program (Fast Track or Phase II Enhancement) “on this Phase II”? High percentages of both groups expressed satisfaction with their decision to use those programs (90 percent of the Fast Track group and 95 percent Phase II Enhancement group). These questions about both Fast Track and Phase II Enhancement were only asked of participants in those programs. The next series of questions elicited opinions from firms who had not or would not in the future participate in one of these programs. 97 Ibid.

OCR for page 28
SURVEY ANALYSIS 69 Dissatisfied Satisfied 100 90 Percent of Respondents 80 70 60 50 40 30 20 10 0 Phase II Enhancement Fast Track FIGURE 2-23 Customer satisfaction with programs: Decision to use Fast Figure 2-23.eps Track or Phase II Enhancement. SOURCE: NRC Project Survey 2006, Questions 28 and 30. See Appendix C. 2.2.9 Reasons for Not Submitting Fast Track and Phase II Enhancement Proposals The following information was determined from questions on the firm survey.98 Fast Track Firms were asked if they had ever submitted a Fast Track proposal.99 Thirty-eight percent of those responding had submitted a Fast Track proposal. Of those who had submitted a Fast Track proposal, 38 percent said that they were likely to do so in the future. However, firms did not express strong opinions in responding as to whether they would submit future Fast Track proposals. Nineteen percent of all respondents said yes; one-third said no; and the rest did not answer this question. Ten percent of the firms that had submitted for Fast Track in the past said that they would not do so in the future. Firms that had not submitted, or were not likely to submit a Fast Track proposal, were asked to identify why. Their responses are shown in Figure 2-24. 98 The project survey is not the basis for categorizing a firm as belonging to Fast Track, Phase II Enhancement, or control groups. This is because a given firm (1) could have been surveyed in more than one award category and/or (2) could have award(s) in another category but not sampled in that category. 99 NRC Firm Survey 2006, Question 15. See Appendix B.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 70 Difficulty obtaining timely third-party funding Insufficient benefits provided by Fast Track Other Concern about dilution of equity / control Bureaucratic requirements of Fast Track Government sponsor opposed Fast Track Dissatisfied with prior Fast Track experience 0 10 20 30 40 50 60 70 80 Percent of Respondents FIGURE 2-24 Reasons for Figure 2-24.epsTrack proposals not submitting Fast SOURCE: NRC Firm Survey 2006, Question 16. See Appendix B. Difficulty in finding third-party funding in time to submit a Fast Track application100 was by far the primary reason for not submitting or planning to submit. Firms that responded “other” offered a variety of reasons. The most frequent response (4 of 29 who responded other) was lack of knowledge or understanding of Fast Track. The second most frequent response (3 of 29) was that the firm was no longer eligible for SBIR. Only one other response, dealing with lack of commercial maturity of the technology, was given by more than one firm. Phase II Enhancement The firms were asked the same questions about participation in Phase II Enhancement.101 Fifty-one percent of firms reported that they had submitted a Phase II Enhancement proposal. Of these, 61 percent reported that they were likely to do so in the future. Although many firms indicated that they were likely to submit future Phase II Enhancement proposals, again many expressed no opinion. Forty-six percent of all respondents said that they were likely to submit future Phase II Enhancement proposals; eight percent said that they were 100 Although Fast Track Phase II proposals are not due for six or more months (component dependent) after the Phase I award, Fast Track applications are due 150 days after the Phase I award. 101 NRC Firm Survey 2006, Question 17. See Appendix B.

OCR for page 28
SURVEY ANALYSIS 71 unlikely; and the rest did not answer this question. Six percent of the firms that had submitted for Phase II Enhancement in the past reported that they would not do so in the future. Firms that had never submitted for Phase II Enhancement or which said they would not do so in the future were asked to identify why. Their responses are shown in Figure 2-25. Because some components require that third-party funding be provided by acquisition programs, the response dealing with such funding was split to differentiate between private sector and acquisition program funding. Many respondents (26 of 38) who identified difficulty in obtaining acquisition program funding also identified difficulty in obtaining private sector funding. For firms reporting “other,” the most frequent response (over half of the 32 firms who responded “other”) was lack of knowledge or understanding of Phase II Enhancement. This may indicate a need to better publicize the program. The second most frequent response (5 of 32) was that the firm was no longer eligible for SBIR. No “other” response was given by more than one firm. 2.2.10 Dilution of Ownership Some SBIR contractors may be reluctant to seek third-party financing, particularly from venture capital companies, fearing that they will lose control of their firms. Others may welcome such cash infusions, preferring to have partial control over a potentially large firm to full control of a small one.102 Some in this latter group may intend to sell their interests as the firm gets large and roll the profits into starting a new firm. Those leery of equity funding are often heavily involved in advancing technology and less interested in production. They are afraid that outside investors would change the fundamental nature of the firm. The business expertise that venture capital insists on putting in place (if not already there) creates an environment likely to produce commercial success, but such success may not be the principal goal of the owners.103 Third-party investors may also be larger firms who want to merge with or partially or completely own the SBIR firm after the award. Finalized agreements and ongoing negotiations are shown in Figure 2-26. 102 Other sources of funding include family, angel investors, and state programs, though the role of these diverse sources changes as the firms evolves from the seed and early stages of development through to the later states. 103 For a qualitative study of entrepreneur perspectives on conceptualizing and starting successful ventures, see, F. G. Crane and J. Sohl, “Imperatives for Venture Success: Entrepreneurs Speak,” The International Journal of Entrepreneurship and Innovation, May 2004, pp. 99-106. For an overview of the growth, characteristics, and challenges of the venture capital industry in the United States, see Paul Alan Gompers and Joshua Lerner, The Venture Capital Cycle, 2nd Edition, Cambridge, MA: MIT Press, 2004.

OCR for page 28
REVISITING THE DEPARTMENT OF DEFENSE SBIR FAST TRACK INITIATIVE 72 Difficulty obtaining third-party private-sector funding Difficulty obtaining DoD Acquisition Program funding Insufficient benefits provided by Phase II Enhancement Other Concern about dilution of equity / control Bureaucratic requirements of Phase II Enhancement Government sponsor opposed Phase II Enhancement Dissatisfied with prior Phase II Enhancement experience 0 10 20 30 40 50 60 Percent of Respondents FIGURE 2-25 Reasons for not submitting Phase II Enhancement proposal. Figure 2-25.eps SOURCE: NRC Firm Survey 2006, Question 18. See Appendix B. Merger negotiations 40 Partial sale negotiations 35 Percent of Phase II Reporting Sale negotiations 30 Finalized merger Finalized partial sale 25 Finalized sale 20 15 10 5 0 Fast Track Phase II Control Group Enhancement FIGURE 2-26 Dilution of firm ownership SOURCE: NRC Project SurveyFigure 2-26.eps See Appendix C. 2006, Question 12.

OCR for page 28
SURVEY ANALYSIS 73 Fast Track The figure indicates that overall FT firms were much more likely than control group firms to be engaged in equity-related activities. Given the need for matching funds this is of course not surprising. Figure 2-23 portrays the “cost” to firm owners of Fast Track firms. To obtain third-party funding, they may be limiting their personal share of the potential gain from their innovation, by selling a share of their firm. But they may also be increasing the ultimate gain from the innovation by infusing cash and business expertise at the critical point in development. Fourteen percent of the Fast Track awards have resulted in finalized agreements for sale of ownership, partial sale of ownership, or merger. Negotiations are ongoing for another 22 percent. Whether this is a cost or an opportunity is very much a personal evaluation. This compares with one percent of firms (one firm) reporting a finalized agreement, in this case for partial sale of the firm. Phase II Enhancement The impact of equity-related activities is lower for Phase II Enhancement firms, where only 6 percent report finalized activity in sale or merger. This is still considerably greater than for the control group.