. "2 Findings and Recommendations." An Assessment of the Small Business Innovation Research Program at the Department of Defense. Washington, DC: The National Academies Press, 2009.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
An Assessment of the SBIR Program at the Department of Defense
2
Findings and Recommendations
NRC STUDY FINDINGS
The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country.
A substantial percentage of SBIR projects at DoD commercialize.1
The NRC Phase II Survey, which was sent to all firms with Phase IIawards from 1992 to 2002, provides evidence of substantial, if highlyskewed, commercialization.2
Nearly half (46 percent) of respondents indicated that the surveyed SBIR project had reached the marketplace (i.e., they reported more than $0 in sales and licensing revenues from the project by May 2005, which is the closing date of the survey).3
1
All data in this section are drawn from the NRC Phase II Survey, unless otherwise stated. Commercialization refers here to the extent to which projects generate outcomes that have market value. Commercialization in the context of DoD also refers to the take-up of projects within DoD, often (but not always) in the context of Phase III funding from non-SBIR resources. This aspect of commercialization is taken up in Section 4.3.
2
See Appendix B for a detailed description of the survey, response rate, and related issues. For DoD, the response rate was 42 percent of the awards contacted. See also National Research Council, Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books.nap.edu/catalog.php?record_id=11097#toc>.
3
See Figure 4-1 (NRC Phase II Survey, Question 1 and Question 4).
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole. Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 23
An Assessment of the SBIR Program at the Department of Defense
2
Findings and Recommendations
NRC STUDY FINDINGS
The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country.
A substantial percentage of SBIR projects at DoD commercialize.1
The NRC Phase II Survey, which was sent to all firms with Phase II awards from 1992 to 2002, provides evidence of substantial, if highly skewed, commercialization.2
Nearly half (46 percent) of respondents indicated that the surveyed SBIR project had reached the marketplace (i.e., they reported more than $0 in sales and licensing revenues from the project by May 2005, which is the closing date of the survey).3
1
All data in this section are drawn from the NRC Phase II Survey, unless otherwise stated. Commercialization refers here to the extent to which projects generate outcomes that have market value. Commercialization in the context of DoD also refers to the take-up of projects within DoD, often (but not always) in the context of Phase III funding from non-SBIR resources. This aspect of commercialization is taken up in Section 4.3.
2
See Appendix B for a detailed description of the survey, response rate, and related issues. For DoD, the response rate was 42 percent of the awards contacted. See also National Research Council, Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books.nap.edu/catalog.php?record_id=11097#toc>.
3
See Figure 4-1 (NRC Phase II Survey, Question 1 and Question 4).
OCR for page 24
An Assessment of the SBIR Program at the Department of Defense
Of the 420 projects reporting some sales, just under 1 percent reported sales greater than $50 million, another 9.2 percent of projects reporting some commercialization, indicated sales between $5 million and $50 million.4
In addition, 17.6 percent of respondents reported sales by licensees of their technology, with three reporting licensee sales of greater than $50 million.5,6
For projects that have received sales, survey responses indicate that 87.6 percent of first sales occurred within 4 years of the Phase II award date.7 Interviews and cases, however, support the view that the bulk of sales will be realized in the longer run—that is, beyond the date of first sale.
These figures, while positive, necessarily reflect the concentration and skewed outcomes often associated with early-stage funding and the special challenge of the procurement process. The figures also understate, perhaps substantially, the amount of commercialization ultimately to be generated from the funded projects. It is important to recognize that these data constitute only a snapshot of sales and licensing revenues, as of May 2005. Projects completed in more recent years will continue to generate revenues well into the future. Consequently, the data aggregated for the May 2005 snapshot necessarily under-reports the eventual return from the SBIR Phase II awards that were made during the latter part of the study period (1992–2002).8
4
See Figure 4-2 (NRC Phase II Survey, Question 4).
5
This type of “skew”—in which a majority of projects fail or are minimally successfully while a small proportion generates large revenues—is typical of early-stage finance and has been noted in previous Academy research. See National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000. See also Joshua Lerner, “Public Venture Capital: Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, DC: National Academy Press, 1999.
6
NRC Phase II Survey, Question 4.
7
Ibid.
8
The total eventual return from these awards is estimated to be approximately 50 percent higher than the data captured at the time of the survey. (For an explanation of the methodology underlying this analysis, see Chapter 4.) This suggests that the actual sales and licensing revenues that will in the end be generated by projects funded during the study period, on average, are approximately $2.2 million, and about $5.6 million for each project that did report some sales or licensing revenues. For DoD as a whole, the SBIR Program Manger Michael Caccuitto, reports that the amount of commercialization generated from SBIR projects now leads the total amount spent on SBIR, with about a 4-year lag from the year of Phase II award. See National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007. This corresponds with the findings of the 1992 GAO report’s assessment of commercialization, which found that not enough time had elapsed since the program’s inception for projects to mature. See U.S.
OCR for page 25
An Assessment of the SBIR Program at the Department of Defense
Success in attracting further research funding for ongoing development offers evidence indicating that a project is on the path to commercialization, even if no sales have yet been made.
Over one-quarter of projects that received additional funding reported the acquisition of additional funds from other federal sources and 13.2 percent reported funding from other companies.9 This suggests significant interest in these projects—not least from DoD, and possibly also among the prime contractors (a likely source of funding from “other companies”).
Venture capital is not widely available to companies primarily focused on the Defense market. Hurdles associated with regulations in federal acquisition, and the limited size of many defense markets tends to limit venture funds’ interest in the DoD market. Only 30 projects—3.8 percent of respondents with some additional funding—reported receiving venture capital,10 although the average VC investment that was received is much higher than the average investment received in each other category, at more than $5 million per project.11
Additional SBIR awards are a further signal of commercial potential.
Given that SBIR is a highly competitive program, the acquisition of related SBIR awards also suggests that a project is moving along the development path toward commercialization, not least because commercialization potential has become a significant component in the decision to make an SBIR award. 43.5 percent of respondents indicated that they had received at least one additional related SBIR award.12
SBIR is in broad alignment with the needs of the DoD agencies and components.
DoD’s SBIR program has contributed to significant enhancements of its mission capabilities.
A central mission of the DoD SBIR program is to use the inventiveness of small companies to solve DoD’s technical problems, and to develop new technologies that can be applied to the weapons and logistics systems that are eventually used by the Armed Forces.13
General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED–92–32, Washington, DC: U.S. General Accounting Office, 1992.
9
Derived from NRC Phase II Survey, Question 23. See also Table App-A-37.
10
Ibid.
11
NRC Phase II Survey, Question 23. See also Table App-A-37. Similar to sales, the amount of venture funding is skewed with only eight projects reporting $5 million or more in venture funding.
12
See Table 4-12.
13
Interviews with SBIR program managers.
OCR for page 26
An Assessment of the SBIR Program at the Department of Defense
BOX 2-1
ArmorWorks, Inc.—Body Armor in Iraq
Technologies developed from SBIR-funded research efforts were used in the design of Small Arms Protective Inserts (SAPI) Body Armor Plates used in the Interceptor Vest currently being worn by U.S. service men and women in the Middle East. ArmorWorks has been awarded more than $50 million in contracts from the Army and Marines to produce SAPI plates for body armor, making them a leading producer for the U.S. military. To date, some 350,000 SAPI plates have been produced for the Department of Defense.
ArmorWorks also manufactures vehicle armor. The company’s SBIR research contributed to the design of HMMWV and add-on armor kits for trucks currently in use in the Middle East. The vehicles armor produced by ArmorWorks has a number of valuable features for the battlefield, including easy installation (requires no vehicle modification or special tools) and field configurability (contains simplified installation to allow for reconfiguration for specific missions). ArmorWorks recently received another $30 million contract from the U.S. Defense Logistics Agency to produce and deliver Kevlar inserts to protect U.S. troops in Iraq and Afghanistan against small arms fire.
SOURCE: DoD SBIR Success Stories, <http://www.dodsbir.net/SuccessStories/armorworks.htm>.
Improved mission capabilities in the context of DoD relate to maintaining technological dominance in battle space conditions, increased responsiveness to new, unexpected situations, such as responding to improvised explosive devices (IEDs), and reductions in the cost of operations and support systems.
DoD SBIR program managers also speak favorably about the creativity of the small- and medium-sized firms that comprise the SBIR community.14
DoD appears to be making a concerted effort to ensure that SBIR projects are aligned with the needs of the weapons system managers.15
The topic development process has since 1999 incorporated important elements specifically designed to align acquisitions needs and SBIR topics.16
14
The agency’s Technical Point of Contacts (TPOCs) were surveyed by the NRC. See NRC Project Manager Survey in National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009.
15
A number of initiatives in this direction are described in Chapter 5: The Phase III Challenge.
16
See Section 6.2.
OCR for page 27
An Assessment of the SBIR Program at the Department of Defense
FIGURE 2-1 Reported Phase III contract awards value 1999–2005.
SOURCE: John Williams, Navy, April 7, 2005.
According to DoD managers, 60 percent of topics published in the SBIR solicitation are now sponsored or otherwise supported by acquisition agencies.17
Based on DoD data, identified Phase III contracts18 now total more than $450 million19 annually, and have been growing rapidly in recent years.
SBIR offers an unusual degree of execution year flexibility and a short
17
The Navy finds that 80 percent of its topics are acquisition linked. Comments by Dr. Holland and Mr. Caccuitto at the NRC Phase III Symposium, June 2005. See National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
18
Many Phase III contracts are not captured effectively by the DD350 reporting system, which undercounts the number and size of Phase III contracts awarded, to a varied degree at different DoD components. See Chapter 5. In addition, commercialization occurring intraindustry, at a subcontract level, or in the commercial marketplace (and then perhaps finding its way to DoD) is not measured by this data source. The DD350 has been replaced by the Federal Procurement Data System—Next Generation (FPDS-NG) as the system tracking, and thus characterizing, all prime contract actions. This system is now standard for the federal government, with a small number of exceptions.
19
$450 million is based on the DD 350: however, the 2005 DoD Annual SBIR Report to SBA lists by contract $565 million in known Phase III contracts and that listing is considered to be incomplete. It includes only contracts known to the SBIR program managers. It should be kept in mind that these contracts are for direct sales to DoD or direct further DoD R&D funding. Sales and further R&D from DoD primes are not included. The NRC Phase II Survey and the DoD Commercialization data indicate the DoD sales and funding are actually less than half of the total commercialization from DoD SBIR.
OCR for page 28
An Assessment of the SBIR Program at the Department of Defense
planning horizon, permitting the program to rapidly address urgent mission needs.
SBIR-supported innovations have contributed to enhanced U.S. combat capabilities, and provided technological solutions to meet sudden, unexpected challenges to our military. Indeed, the high degree of flexibility characteristic of small firms means that SBIR has provided DoD with an increased number of suppliers capable of quickly responding on short notice to unexpected battlefield situations.20
For example, the Navy has taken advantage of this flexibility to issue a “quick response IED topic” in 2004, and made 38 Phase I awards within 5 months of topic development. These developed into 18 Phase II awards, and results from these will be available in 2006–2007. The first prototypes were expected in Iraq in the Fall of 2006.
By contrast, most RTD&E accounts require considerable forward planning.
SBIR increases the number of potential suppliers for new technologies, and also creates new opportunities for these firms to partner together in new undertakings.
Used effectively, SBIR can act as a low-risk, low-cost technology probe and a search tool for finding new technology suppliers. It has helped DoD personnel learn about new technologies, new applications, and a new set of high-tech firms with whom they would not otherwise have contact.21
The laws governing the SBIR program permit the use of sole source procedures when federal agencies acquire technologies developed with SBIR funding. This allows substantially faster acquisition than through standard channels and acts as a powerful incentive for SBIR firms and their partners.
The quality of SBIR-funded research is broadly comparable to that of other non-SBIR research according to the NRC Project Manager Survey.
20
See, for example, the case of ArmorWorks, Inc., in Box 2-1.
21
Firms express this outcome as follows: Had it not been for SBIR, their business with DoD services or agencies would not have developed. Services likely would have stayed with their pre-existing sources of supply. Program managers are normally too busy administering multiple contracts to search out or respond attentively to new sources of technology. Their propensity is to hire a contractor to solve problems rather than seek out the most technologically innovative performer. The SBIR program requires that program managers become involved with small firms, to look at technical options, and to allow for increased competition in the selection of R&D performers.
OCR for page 29
An Assessment of the SBIR Program at the Department of Defense
In the NRC Project Manager Survey, 53 percent of Technical Points of Contact (TPOC) respondents indicated that the specific SBIR project identified in the survey produced results that were useful to them and that they had followed up on this work with other research. SBIR projects are normally part of a wider portfolio of research responsibilities handled by TPOCs.22
The NRC Project Manager Survey also indicated that the quality of SBIR funded research is comparable to non-SBIR research they manage. Normalized survey scores indicate that the quality of SBIR research is equivalent to that of other research at DoD.23
The Department has devoted considerable recent effort to strengthen the critical connection between SBIR and the acquisition programs through Phase III. While this focus is to be commended, the Phase III process can be considerably widened and improved.
A striking aspect of SBIR Phase III at DoD is the extraordinarily uneven character of outcomes and activities between services, and between components within services.
Notably, Phase III transitions at PEO SUBS account for approximately 86 percent of all Navy Phase III contracts, and Navy in turn accounts for about 70 percent of all DoD Phase III contracts, as captured by the DD350 forms completed by contracting officers.24
This skew partly results from the additional effort made by Navy to ensure that DD350 forms are completed and accurately reflect SBIR contributions, which in turn reflects different views of the importance of supporting SBIR, as evidenced by the amount of resources, staff, and funding that services and components allocate for program support.
At some components, such as Navy, senior management recognizes the potential value of SBIR and has supported extensive efforts to build effective bridges between SBIR and the acquisition programs. At other components, efforts have been less well supported, and on the basis
22
See the related discussion in Section 4.3.1.2.
23
The scores were normalized scores by removing the outliers in the top and bottom 5 percent of scores. Statistical procedures often assume that the variables are normally distributed. A significant violation of the assumption of normality can seriously increase the chances of a Type I (overestimation) or Type II (underestimation) error. Nonnormality can occur in the presence of outliers (scores that are extreme relative to the rest of the sample). Removing the outliers can improve the normality of the distribution. See C. M. Judd and G. H. McClelland, Data Analysis: A Model-Comparison Approach. San Diego, CA: Harcourt Brace Jovanovich, 1989.
24
See the discussion in Section 5.2, including Figure 5-1.
OCR for page 30
An Assessment of the SBIR Program at the Department of Defense
of the data presented in Chapter 5, they appear to be considerably less effective.
There is no effective and comprehensive tracking system within DoD to follow SBIR-funded technologies to their final outcome.
The Company Commercialization Report is self-reported data, and must be updated only when a company applies for a new DoD SBIR.
The agency’s DD350 reporting system may substantially undercount Phase III awards, as contracting officers must be specifically trained to capture this data correctly.25
The lack of a reliable and effective tracking system for SBIR awards that would identify follow-on funding sets back efforts to assess the impact of the program and to document its successes.
SBIR awards made by DoD support small businesses in a number of important ways.
SBIR awards have had a substantial impact on participating companies.
Company Creation. Just over 25 percent of companies responding to the NRC Firm Survey indicated that they were founded entirely or partly because of a prospective SBIR award.26
The Decision to Initiate Research. Only 13 percent of DoD project respondents thought that their project would “definitely” or “probably” have gone ahead without SBIR funding. Over two-thirds (about 70 percent) thought they definitely or probably would not have initiated the research; most of those who anticipated that their project would have gone ahead without the award acknowledged the likelihood of substantial delays without the award.27
Company Growth. Almost half (48 percent) of the respondents in-
25
There are multiple limitations to current systems for tracking SBIR awards. The DD350 reporting system can be used to extract some SBIR data; however, the system was not designed to gather SBIR award data, and is used differently by the services and agencies. For example the Navy reports Phase II Enhancements as Phase III, while the other services report such awards as Phase II. The Phase I and Phase II data in the DD350 does not match the SBIR budget or DoD Annual Report to SBA. The Annual Report to SBA, prepared by the DoD SBIR program managers, accounts for the budgeted SBIR funding. The SBIR program managers also have less knowledge of the Phase III awards, since these are made with funds that are not under control of the SBIR program.
26
See Table 4-15 (NRC Firm Survey, Question 1). Data reported in Table 4-15 are for firms with at least one DoD award. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA).
27
See Figure 4-10 (NRC Phase II Survey, Question 13).
OCR for page 31
An Assessment of the SBIR Program at the Department of Defense
FIGURE 2-2 New winners at DoD.
SOURCE: NRC Phase II Survey.
dicated that more than half the growth experienced by their firm was directly attributable to SBIR.28
Partnering. SBIR funding is often used by small firms to gain access to outside resources, especially academic consultants and, often, to seek company partners.
SBIR Awards Attract Participation by New Firms.
FY2005 data from DoD show that 29 percent of Phase I awards went to firms that had not previously won a DoD Phase II SBIR award.
DoD data indicate that a further 50 percent of awards went to companies with five or fewer previous Phase II awards. Only 13 percent of awards went to companies with at least six Phase II awards.29
Data from the Naval Air Systems Command (NAVAIR) show that about half of all its Phase I contracts go to companies that have never won an SBIR award from NAVAIR before. In addition, about 40–45 percent of Phase II contracts go to newly participating firms.30
Multiple SBIR awards serve multiple objectives for firms and agencies. A small number of companies receive multiple awards. A few companies have
28
See Table 4-16.
29
Michael Caccuitto and Carol van Wyk, “Enhancing the Impact of Small Business Innovation Research (SBIR) Program: The Commercialization Pilot Program,” Presentation, September 27, 2006.
30
See Section 3.2.2.
OCR for page 32
An Assessment of the SBIR Program at the Department of Defense
received more than 100 awards at DoD (over 20 years), and are higher in commercialization (on average).31
The myth of the linear innovation model: The program has often encountered criticism with regard to firms—sometimes labeled SBIR “mills”—that are described as winning large numbers of awards but that achieve lower levels of commercialization than firms with significantly fewer awards.32 Implicit in this view of multiple-award winners is the simple linear model of the program. In reality, multiple awards serve a variety of functions, as noted below.
Role of multiple awards. More than one award is often required to develop a technology or firm capability, quite apart from the commercialization of a complete product. These needs reflect the complex and differing characteristics of firms.
Diversity of firm objectives. Reflecting the diversity of the program’s objectives and of the participants, some firms approach the SBIR award process at different stages of development and with different objectives. Some firms are developing technology concepts; some firms see their vocation as contract research organizations; others actively seek to develop commercial products, either for public agencies or for the marketplace.33
Diversity in firm strategy. For example, investigator-led firms, limited in size and focused on a single concept may seek multiple awards as they advance research on a promising technology.34 For firms that
31
It is important to keep in mind the difficulties in tracking companies over time. Companies regularly change names, locations, even employer identification numbers, which makes tracking them across time within the DoD awards database difficult.
32
See the 1992 GAO report, U.S. General Accounting Office, Small Business Innovation Research Program Shows Success But Can Be Strengthened, RCED–92–32, op. cit. That report focused on firms that had received twenty or more awards and on a single program metric, that is, commercialization. Congressional legislation later fixed fifteen awards in a five-year period as the level where efforts to commercialize should be taken into account.
33
See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Program in the Southwestern and Mountain States,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., p. 151. The author describes the incremental nature of technical advance, which sometimes necessitates several awards. See also John T. Scott, “An Assessment of the Small Business Innovation Research Program in New England: Fast Track Compared with Non-Fast Track,” in National Research Council, The Small Business Innovation Research Program: An Assessment of the Department of Defense Fast Track Initiative, op. cit., p. 109 for a discussion of Foster-Miller, Inc.
34
Ibid. The mirror image of this approach is the program manager who makes several awards for similar technologies among different companies. In fact, it is not uncommon to have multiple awardees on the same Phase I topic. For an example, see the Navy’s SBIR Web site selections page for their FY-06.1 awardees at <http://www.navysbir.com/06_1selections.html>. This page not only shows
OCR for page 33
An Assessment of the SBIR Program at the Department of Defense
carry out research as a core activity, success is often measured in multiple contract awards.
Addressing agency missions. Some firms, mainly at DoD, have won large numbers of awards over the life of the program. Yet, even with many awards, there is nothing intrinsically wrong with a process that makes high-quality research available to the department at relatively low cost. Some of this research is intrinsically noncommercial, but may have considerable value.35
Identifying dead ends. Inexpensive exploration of new technological approaches can be valuable, particularly if they limit expenditure on technological dead ends. For research oriented firms, the key issue is the quality of the research and its alignment with service and agency needs.36 Each of the seven most frequent winners, who have received over 100 Phase II awards since the program inception, has a large number of researchers who submit proposals. The high number of quality proposals can produce a high number of awards. Some successful applicants use spin-off firms to commercialize the results of their SBIR awards.
Providing solutions. In some cases, firms respond to an agency solicitation and “solve” the problem, provide the needed data, or propose a solution that can then be adopted by the agency with no further “commercialization” revenues for the firm.37
several awardees for each topic, but if you click on the “Details” link, you can see the differences in companies’ approaches to the topics.
35
To secure additional awards, a small company has to submit its proposal for follow-on research through the regular review process. These awards are relatively small in amount—the normal Phase I and Phase II awards would total $850,000. As a point of comparison, the top three U.S. prime contractors in 2007 garnered over $20.5 billion in defense revenues. See Washington Technology, “Top 100 Federal Prime Contractors: 2004,” May 14, 2007.
36
For example, Foster-Miller, the most frequent SBIR award winner, provided armor for Humvees and aircraft and developed robots for use in Iraq to identify roadside improvised explosive devices. (This company has since been acquired, making it ineligible to participate in SBIR. See Financial Times, “Qinetiq set to make its first US acquisition,” September 8, 2004.) Their LAST® Armor has had sales in excess of $170 million.
37
There are cases where a small business successfully completes the requirements and objectives of a Phase II contract, meeting the needs of the customer (e.g., by delivering specialized software or hardware), without gaining additional commercialization revenues. For example, Aptima, Inc., designed an instructional system to improve boat-handling safety by teaching the use of strategies that mitigate shock during challenging wave conditions. A secondary goal was to demonstrate how an innovative learning environment could establish robust skill levels while compressing learning time. Phase I of the project developed a training module, and in Phase II, instructional material, including computer animation, videos, images, and interviews were developed. The concept and the supporting materials were adopted as part of the introductory courses for Special Operations helmsmen with the goal of reducing injuries and increasing mission effectiveness.
OCR for page 41
An Assessment of the SBIR Program at the Department of Defense
bility, etc.) from these companies for all their previous SBIR awards, including those at other agencies. It also captures information on firm size and growth since entering the SBIR program, as well as the percent of annual revenue derived from SBIR awards. These historical results of prior awards are then used in proposal evaluation.
Non-DoD agencies should consider adapting both this approach and the DoD technology and contributing to the DoD database. This would provide a unified tracking system. Adaptations could be made to track additional data for specific agencies, but this would provide a cost-effective approach to enhance data collection on award outcomes.
Multiple agencies. Gap-reduction strategies. The agencies have, to different degrees, recognized the importance of reducing funding gaps. While details vary, best practice would involve development of a formal gap-reduction strategy with multiple components covering application, selection, contract negotiation, the Phase I–Phase II gap, and support after Phase II.
DoD Phase II Enhancement (and NSF Phase IIB). The matching-fund approach adopted by NSF for Phase IIB and DoD for Phase II Enhancement might be explored at other agencies. The NSF matching requirement represents an important tool for helping companies to enter Phase III at nonprocurement agencies. The DoD funding match by acquisition programs provides a transition link into Phase III contracts with the agency.
DoD-Navy. Technology Assistance Program. The Navy has developed the most comprehensive suite of support mechanisms for companies entering Phase III, and has also developed new tools for tracking Phase III outcomes. These are important initiatives, and other components and agencies should consider them carefully.
recent reviews represent a positive effort to connect data and analysis to practice.56
Phase III results remain uneven across the services and among components within the services.
This suggests that other components and other services could improve the performance of their programs.
Interviews suggest that some elements of the department have not fully integrated SBIR within their own program missions and have not
56
See also a recent RAND Report requested by DoD: Bruce Held, Thomas Edison, Shari Lawrence Pfleeger, Philip S. Anton, and John Clancy, Evaluation Recommendations for Improvement of the Department of Defense Small Business Innovation Research (SBIR) Program, Santa Monica, CA: RAND Corporation, 2006.
OCR for page 42
An Assessment of the SBIR Program at the Department of Defense
provided SBIR with the resources and management attention needed to maximize its effectiveness.57
The 2005 NRC Symposium on SBIR Phase III contributed to the awareness of the SBIR program’s potential, the challenge that promising products face in the Phase III transition, and the need for additional efforts to “team” across agencies, with SBIR program managers, Program Executive Officers, and prime contractors.58
Prime contractors are taking a positive approach towards the SBIR program.
Increased interest in SBIR. As the program has grown in size and performance, it has garnered greater attention from the DoD upper management and, importantly, the prime contractors.
There is considerable evidence that prime contractor interest in—and engagement with—the SBIR program has been growing rapidly in recent years (see Chapter 6). Concretely, this is reflected in growing contractual linkages.59
Interest has been followed by action. This increased focus on the prospective contributions of the SBIR program by the prime contractors appears to represent a significant positive endorsement of the contributions of the program.
Steps taken by the prime contractors to integrate SBIR within their
57
Some DoD program managers see the SBIR program as a “tax,” that is an unwarranted allocation of funds to small business, one that hinders effective R&D program management and one that is time consuming to manage and without funds to cover the cost of management. For a more positive view of the program, see the discussion of the Navy program in the Findings section. The recommendations address program perception and incentives.
58
Following the National Academies meeting on the SBIR commercialization challenge, the Senate Small Business & Entrepreneurship Committee proposed legislation that established a Commercialization Pilot Program. See Section 252 of the 2006 National Defense Authorization Act. The bill was passed in bipartisan spirit by the Senate Committee on Small Business & Entrepreneurship (SBE) under the leadership of the Committee Chair, Olympia Snowe (R-ME) and Ranking Member, John Kerry (D-MA). Further reflecting the growing appreciation of the program’s role and the increased focus on Phase III, Dr. Finley has described the SBIR program as a means of accelerating innovation and putting better equipment into the hands of the war-fighter. See remarks by Deputy Under Secretary of Defense for Acquisition and Technology Finley at the Small Business Technology Coalition Conference on September 27, 2006.
59
Raytheon, for example, estimates that the value of technology leveraged through SBIR jumped from $3.8 million in 2004 to $11.6 million in 2005, and looks set to grow as rapidly in 2006. Raytheon is involved with 36 Phase I projects, 17 Phase II projects, 4 Phase III projects (with three more in the works), and has been a subcontractor on other projects. See Lani Loell, SBIR Program Manager Raytheon Integrated Defense Systems, Presentation to SBTC SBIR in Transition Conference, September 27, 2006, Washington, DC.
OCR for page 43
An Assessment of the SBIR Program at the Department of Defense
strategic roadmaps reveal that they see the program contributing to technological innovations that further the Defense mission.60
At the Academies’ Phase III conference, representatives of prime contractors stated that there was already a substantial amount of prime involvement with the SBIR program. Moreover, several of the primes affirmed that they had made significant efforts to increase their levels of involvement.
For example, Boeing had recently decided to increase its emphasis on SBIR.
Similarly, at Raytheon, some divisions (e.g., Integrated Defense Systems) had formal working arrangements with SBIR for several years.
NRC STUDY RECOMMENDATIONS
As noted in the Findings section above, the Department of Defense has an effective SBIR program. The recommended improvements listed below should enable the DoD SBIR managers to address the four mandated congressional goals in a more efficient and effective manner.
Improve the Phase III transition. DoD should continue to expand its work on improving the Phase III transition (the transition from SBIR-funded Phase I and Phase II research to commercialization—especially testing and evaluation funded by other DoD sources). It is important to recognize that the transition of new technologies is a complex process requiring teaming across areas of responsibility, additional resources, and often coping with some element of additional risk.61Areas for possible action include:
Aligning incentives.
For the SBIR program to achieve its full potential, better incentives are required. Expansion of positive incentives for program officers to utilize the SBIR program for their own research needs, beyond the current requirements for their involvement in topic development.
Management needs to improve incentives so that acquisitions officers perceive reduced risks and enhanced benefits from participating in the program.
Increasing resources. SBIR managers need greater resources to “match” program funds to encourage uptake. In addition to increased Phase III
60
See Section 5.4.4 and the discussion in Chapter 6.
61
See the discussion of these problems and potential remedies in the section on Phase III Transition in this volume. See also National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
OCR for page 44
An Assessment of the SBIR Program at the Department of Defense
SBIR funding, linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition.
Developing an evaluation culture. Agency and service managers should have effective data collection and analysis as performance metrics.
Involving acquisition officers. Active participation by acquisition officers is key to successful Phase III transitions.
Acquisition officers control the funding, and their involvement is important for successful commercialization of SBIR technologies. A cultural shift in program participation and use seems to have occurred at Navy once Program Executive Offices (PEO) became active champions of SBIR involvement in acquisitions.
Senior management support and encouragement, better information flows, improved PEO education about SBIR, and additional incentives for PEOs to use SBIR are all elements of an effective overall program.
Integrating with roadmaps. The long technology development and acquisition cycle for major weapons and logistics systems means that effective Phase III transition requires early integration of SBIR topics and firms into the planning process.62
Linking SBIR with other programs. Linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition.
Improving outreach and matchmaking. There are significant barriers to the flow of information among SBIR firms, prime contractors, and acquisitions offices. Effective transition requires that these barriers be overcome, most likely through implementation of a range of activities, including improved electronic communications methods and matchmaking services like the Navy Opportunity Forum. In particular, efforts should be made, as appropriate, to align the SBIR program with the needs of the prime contractors responsible for the development of major systems.
Connecting with the primes. The growing interest among prime contrac-
62
In the case of the semiconductor industry, the industry perceived early on that problems of coordination could arise with a complex technology, multiple participants, and many ways of proceeding. This realization led to cooperative efforts led by SEMATECH to develop a technology roadmap setting out the relationships among science, technology, and applications as a point of reference for the researchers, technologists, project managers, suppliers, and users involved in and affected by the consortium’s work. As a general approach, roadmaps can advance similar coordination functions in other industry partnerships and in this way contribute to more efficient and more cooperative research. See National Research Council, Government-Industry Partnerships for the Development of New Technologies: Summary Report, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2003.
OCR for page 45
An Assessment of the SBIR Program at the Department of Defense
tors of the SBIR program’s outputs and opportunities for partnering with SBIR companies should be encouraged. Consideration should be given to performance incentives to further encourage development of SBIR supported technologies.
Assessing and expanding commercialization programs. Commercialization programs that provide training, counseling, and networking opportunities should be assessed and, as appropriate, expanded.
DoD should take immediate steps to enhance the perception of the SBIR program’s potential and accomplishments, promoting SBIR as an opportunity.
A key element in the program’s operation is the attitude taken towards the program by the different levels of management in the Defense research and development community.
Where SBIR is seen as an unwarranted intrusion on program management, a “tax” on R&D resources, it is less likely to be effectively aligned with service needs and less likely to have the resources to develop and ultimately insert the results of successful Phase II technologies in weapons and logistics systems and other programs.
When the program is seen as an effective tool to engage the ingenuity of small companies in support of the Defense mission, with shorter lead times and more flexibility, it is much more likely to have its results adopted and incorporated.
In short, there is an element of circularity in developing measures to enhance program effectiveness and management’s guidance and rewards for those managers who use the program effectively. Providing the resources and incentives for managers to see opportunity rather than obligation may well enhance program effectiveness.
DoD should substantially strengthen and expand its evaluation efforts in order to further develop a program culture that is driven by outcomes, data, and internal and external evaluations.
Efforts to identify outcomes should be improved, and evaluations63 should
63
For example, each SBIR award has a DoD technical monitor who serves as the contracting officer’s technical representative (COTR). This individual monitors the contractor’s performance during Phase I and recommends or issues the invitation for Phase II, monitors performance of Phase II, and receives the contractor’s Phase I and II reports. Despite the importance of the TPOC’s role, there is no systematic attempt to use the technical monitor to evaluate the quality of the SBIR efforts and to facilitate Phase III. Since these individuals are not funded separately for SBIR, dissemination of SBIR outcomes and assistance in transition is a function of individual initiative and competing responsibilities. DoD should consider training, funding, and making better use of this valuable asset.
OCR for page 46
An Assessment of the SBIR Program at the Department of Defense
be connected much more directly to program management. More attention should be devoted to the role and contributions of the Contracting Officer’s Technical Representative (COTR). It is important that DoD create the capability to use outcomes data to help assess best practice.
New mechanisms need to be developed that allow for the efficient design, implementation, and subsequent assessment of pilot programs.
Efforts should be increased to make sure that appropriate metrics and benchmarks are adopted and implemented by all units, components, and Services.
DoD should encourage and support the development of a results-oriented SBIR program with a focused evaluation culture.
Effective oversight requires additional staff and funding. Effective management of a data-driven SBIR program requires the regular collection of higher-quality data and systematic assessment. Currently, sufficient resources are not available for these functions. Additional funding should be provided for program management and assessment.
This funding should also be used to provide management oversight, including site visits, improved data collection and analysis, regular reporting, program review, and systematic third-party assessments.
To help foster an active evaluation culture, DoD should consider: Preparing an expanded annual SBIR program report. DoD should
prepare an annual SBIR program report, which gathers all relevant data about awards, outcomes, program activities, and management initiatives. In particular, the Department should publish detailed data annually about Phase III take-up at each service, and at each component within each service, as well as providing information about program initiatives.
Commissioning regular assessments. SBIR programs at both DoD and the individual DoD components should seek to enhance a data-driven management approach, with regular assessment supporting policy development and program management.
Instituting systematic and objective, outside review. The internal assessment program should be supported and supplemented by systematic, objective outside review and evaluation, as envisaged in the reauthorization legislation.
Convening an advisory board. DoD should consider development of a formal advisory board, which would receive the annual program
OCR for page 47
An Assessment of the SBIR Program at the Department of Defense
report and provide its own supplementary review of the report and management practices on an annual basis to senior DoD officials in charge of the SBIR program, or possibly to a subpanel of the Defense Sciences Board.
DoD should consider greater internal review and adoption of best practices.
Such an assessment would identify best practices within DoD and develop mechanisms for encouraging other components to implement these practices within their SBIR programs.
One important example of best practice might be one focused on the Phase III transition at the Navy (see Box 2-3).
Additional research should be undertaken to address, inter alia, three questions related to Navy practice.
What unique factors make the Navy SUBS program successful?
What role is played by initiatives at the service level in supporting the SUBS program?
What elements of that success can best be transferred elsewhere in DoD as best practices to be followed? What changes will be needed to make those transfers successful?
DoD should encourage and support pilot programs that evaluate new tools for improving the program’s overall performance.
Innovation through pilot programs. Making changes initially through pilot programs allows DoD to alter selected areas on a provisional basis; a uniform approach is unlikely to work well for all components of a program that funds highly diverse projects with very different capital requirements and very different product development cycles.
Some possible pilot projects include:
Small Phase III awards. These could be a key to bridging the financing ‘Valley of Death’ that many firms face in converting research to innovation to products.64 NASA for example sometimes provides a small Phase III award—perhaps enough money to fly a demonstration payload—for a technology not ready for a full Phase III. These might
64
See Lewis Branscomb and Philip Auerswald, Between Invention and Innovation: An Analysis of Funding for Early-Stage Technology Development, NIST GCR 02-841, Gaithersburg, MD: National Institute of Standards and Technology, Prepared for the Economic Assessment Office, Advanced Technology Program, November 2002. See also National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
OCR for page 48
An Assessment of the SBIR Program at the Department of Defense
BOX 2-3
Lessons from the Navy Model
Many of the issues we identify with regard to the SBIR program at the Department of Defense have been addressed, with considerable success, by the Navy SBIR program. Keeping in mind the appropriate caveats concerning different agency needs, operating conditions, and cultural traditions, a number of aspects of the Navy program address these concerns. Key features of the Navy program include:
Positive acceptance of the program. Navy PEO’s and program managers increasingly appear to see the SBIR program as a useful tool in meeting mission objectives, as acquisition staff are drawn increasingly into topic development and SBIR project management.
Top management focus. One reason for the positive perception of the program’s utility is that the Navy provides significant management attention, particularly at the program executive officer (PEO) level, to the integration of SBIR into technology development to meet program needs. Strong leadership from the Navy hierarchy emphasizes the potential of SBIR for Navy missions.
Administrative funding and activities. The Navy provides substantial additional funding, now on the order of $20 million per annum, to operate the program. These funds meet a variety of needs ranging from additional professional staff support, funding for the Technology Assistance Program, and resources for the Navy Opportunity Forum that helps match SBIR companies with potential customers.
be combined with milestones or gateways to additional rounds of Phase II funding.
Unbundling larger contracts. Organizing larger contracts into smaller components would tend to open more Phase III opportunities for SBIR firms.65
Redefining testing and evaluation within SBIR. DoD could pilot adoption of a wider view of RDT&E, so that SBIR projects could qualify for limited testing and evaluation funding. That in turn would help fund improvements in readiness levels.
“Spring loading” Phase III, by putting in place Phase II milestones that could help to trigger initial Phase III funding. This could possibly
OCR for page 49
An Assessment of the SBIR Program at the Department of Defense
Emphasis on Phase III funding and process. Navy PEO’s have embraced the challenge of maturing innovative technologies (including SBIR products), which requires serial funding for the many testing, evaluation and demonstration steps that precede acquisition.
Demonstration effects and program integration. The strong Phase III take-up recently demonstrated at Navy suggests that acceptance of the program as a valid and useful component in the Navy’s overall technology development strategy creates a virtuous cycle. Successfully transitioned technologies such as the SAVI logistics tracking system, the DSR sonar, the ACR’s “Silver Fox” reconnaissance UAV, and cost-saving diagnostic technologies provide powerful demonstration effects, underscoring the potential contributions of the program to meet a broad range of Navy needs.
Documented achievement. One of the distinctive features of the programs at Navy is that it successfully documents its accomplishments.a Data from DD350 reports shows that Navy’s Phase III contracts grew from $50 million in 2000 to $350 million in 2005. While these growing achievements may in part reflect unique or superior record keeping as compared with other services, this in itself reflects successful adoption of a data-driven assessment culture. The data gathered provides superior information and improved understanding of the operation and potential of the program.
aOne factor contributing to the Navy’s assessment culture is the tradition of research excellence by the Office of Naval Research, one of the nation’s preeminent federal research funding agencies.
occur in the context of larger, staged, Phase II awards in which additional stages fund more Demonstration and T&E, where non-SBIR funds or resources are leveraged.
Evaluating pilots. DoD should develop a formal mechanism for designing, implementing, and evaluating pilot programs. Pilot programs allow agencies to investigate program improvements at lower risk and potentially lower cost. Effective pilot programs require rigorous design and evaluation, clear metrics for success, and the necessary resources and internal support.
A flexible approach is required. In some cases, pilot programs may require waivers from SBA for activities not otherwise permitted under the SBA guidelines.66 SBA should be encouraged to take a highly flexible view of all agency proposals for pilot programs.
66
In the past, SBA has shown commendable flexibility in allowing agencies to deviate from standard award sizes to accommodate the needs of the technology in question and to devise program innovations, such as the NSF Phase IIB, to provide incentives for commercialization.
OCR for page 50
An Assessment of the SBIR Program at the Department of Defense
Provide additional management resources. To carry out the measures recommended above to improve program utilization, management, and evaluation, the program should be provided with additional funds for management and evaluation.
Effective oversight relies on appropriate funding.67 A data-driven program requires high quality data and systematic assessment. As noted above, sufficient resources are not currently available for these functions.
Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other necessary management activities.
In considering how to provide additional funds for management and evaluation, there are three ways that this might be done:
Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done.
Funds might be drawn from the existing set-aside for the program to carry out these activities.
The set-aside for the program, currently at 2.5 percent of external research budgets, might be marginally increased, with the goal of providing management resources necessary to maximize the program’s return to the nation.68
DoD should take steps to increase the participation and success rates of woman- and minority-owned firms in the SBIR program.
67
According to recent OECD analysis, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent for small programs and 1 percent for large-scale programs. See “Evaluation of SME Policies and Programs: Draft OECD Handbook,” OECD Handbook, CFE/SME(2006)17.
68
Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified; the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year, or roughly double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot. See also the National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.
OCR for page 51
An Assessment of the SBIR Program at the Department of Defense
Encourage participation. Develop targeted outreach to improve the participation rates of woman- and minority-owned firms, and strategies to improve their success rates. These outreach efforts and other strategies should be based on causal factors determined by analysis of past proposals and feedback from the affected groups.69
Encourage emerging talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, especially in the biomedical sciences. This means that many of the woman and minority scientists and engineers with the advanced degrees usually necessary to compete effectively in the SBIR program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. Over time, this talent pool could become a promising source of SBIR participants.
Improve data collection and analysis. The Committee also strongly encourages the agencies to gather and report the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity.
69
This recommendation should not be interpreted as lowering the bar for the acceptance of proposals from woman- and minority-owned companies. Rather it should be seen as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups.
