2
Findings and Recommendations

NRC STUDY FINDINGS

The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country.

  1. A substantial percentage of SBIR projects at DoD commercialize.1

    1. The NRC Phase II Survey, which was sent to all firms with Phase II awards from 1992 to 2002, provides evidence of substantial, if highly skewed, commercialization.2

      • Nearly half (46 percent) of respondents indicated that the surveyed SBIR project had reached the marketplace (i.e., they reported more than $0 in sales and licensing revenues from the project by May 2005, which is the closing date of the survey).3

1

All data in this section are drawn from the NRC Phase II Survey, unless otherwise stated. Commercialization refers here to the extent to which projects generate outcomes that have market value. Commercialization in the context of DoD also refers to the take-up of projects within DoD, often (but not always) in the context of Phase III funding from non-SBIR resources. This aspect of commercialization is taken up in Section 4.3.

2

See Appendix B for a detailed description of the survey, response rate, and related issues. For DoD, the response rate was 42 percent of the awards contacted. See also National Research Council, Assessment of the Small Business Innovation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at <http://books.nap.edu/catalog.php?record_id=11097#toc>.

3

See Figure 4-1 (NRC Phase II Survey, Question 1 and Question 4).



The National Academies | 500 Fifth St. N.W. | Washington, D.C. 20001
Copyright © National Academy of Sciences. All rights reserved.
Terms of Use and Privacy Statement



Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 23
2 Findings and Recommendations NRC STUDY FINDINGS The SBIR program at the Department of Defense is meeting the legislative and mission-related objectives of the program. The program is contributing directly to enhanced capabilities for the Department of Defense and the needs of those charged with defending the country. A. A substantial percentage of SBIR projects at DoD commercialize.1 1. The NRC Phase II Survey, which was sent to all firms with Phase II awards from 1992 to 2002, provides evidence of substantial, if highly skewed, commercialization.2 Nearly half (46 percent) of respondents indicated that the surveyed SBIR project had reached the marketplace (i.e., they reported more than $0 in sales and licensing revenues from the project by May 2005, which is the closing date of the survey).3 1All data in this section are drawn from the NRC Phase II Survey, unless otherwise stated. Com - mercialization refers here to the extent to which projects generate outcomes that have market value. Commercialization in the context of DoD also refers to the take-up of projects within DoD, often (but not always) in the context of Phase III funding from non-SBIR resources. This aspect of com - mercialization is taken up in Section 4.3. 2 See Appendix B for a detailed description of the survey, response rate, and related issues. For DoD, the response rate was 42 percent of the awards contacted. See also National Research Council, Assessment of the Small Business Innoation Research Program—Project Methodology, Washington, DC: The National Academies Press, 2004, accessed at . 3 See Figure 4-1 (NRC Phase II Survey, Question 1 and Question 4). 

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE Of the 420 projects reporting some sales, just under 1 percent reported sales greater than $50 million, another 9.2 percent of projects reporting some commercialization, indicated sales between $5 million and $50 million.4 In addition, 17.6 percent of respondents reported sales by licensees of their technology, with three reporting licensee sales of greater than $50 million.5,6 For projects that have received sales, survey responses indicate that 87.6 percent of first sales occurred within 4 years of the Phase II award date.7 Interviews and cases, however, support the view that the bulk of sales will be realized in the longer run—that is, beyond the date of first sale. These figures, while positive, necessarily reflect the concentration and skewed outcomes often associated with early-stage funding and the special challenge of the procurement process. The figures also understate, perhaps substantially, the amount of commercialization ultimately to be generated from the funded projects. It is important to recognize that these data constitute only a snapshot of sales and licens- ing revenues, as of May 2005. Projects completed in more recent years will continue to generate revenues well into the future. Consequently, the data aggregated for the May 2005 snapshot necessarily under- reports the eventual return from the SBIR Phase II awards that were made during the latter part of the study period (1992–2002). 8 4 See Figure 4-2 (NRC Phase II Survey, Question 4). 5This type of “skew”—in which a majority of projects fail or are minimally successfully while a small proportion generates large revenues—is typical of early-stage finance and has been noted in previous Academy research. See National Research Council, The Small Business Innoation Research Program: An Assessment of the Department of Defense Fast Track Initiatie, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2000. See also Joshua Lerner, “Public Venture Capital: Rationales and Evaluation,” in National Research Council, The Small Business Innoation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, DC: National Acad- emy Press, 1999. 6 NRC Phase II Survey, Question 4. 7 Ibid. 8The total eventual return from these awards is estimated to be approximately 50 percent higher than the data captured at the time of the survey. (For an explanation of the methodology underlying this analysis, see Chapter 4.) This suggests that the actual sales and licensing revenues that will in the end be generated by projects funded during the study period, on average, are approximately $2.2 million, and about $5.6 million for each project that did report some sales or licensing revenues. For DoD as a whole, the SBIR Program Manger Michael Caccuitto, reports that the amount of commercialization generated from SBIR projects now leads the total amount spent on SBIR, with about a 4-year lag from the year of Phase II award. See National Research Council, SBIR and the Phase III Challenge of Commercialization, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2007. This corresponds with the findings of the 1992 GAO report’s assessment of commercialization, which found that not enough time had elapsed since the program’s inception for projects to mature. See U.S.

OCR for page 23
 FINDINGS AND RECOMMENDATIONS 2. Success in attracting further research funding for ongoing develop - ment offers evidence indicating that a project is on the path to com- mercialization, even if no sales have yet been made. Over one-quarter of projects that received additional funding reported the acquisition of additional funds from other federal sources and 13.2 percent reported funding from other companies.9 This suggests signifi- cant interest in these projects—not least from DoD, and possibly also among the prime contractors (a likely source of funding from “other companies”). Venture capital is not widely available to companies primarily focused on the Defense market. Hurdles associated with regulations in federal acquisition, and the limited size of many defense markets tends to limit venture funds’ interest in the DoD market. Only 30 projects—3.8 percent of respondents with some additional funding—reported receiv- ing venture capital,10 although the average VC investment that was received is much higher than the average investment received in each other category, at more than $5 million per project.11 3. A dditional SBIR awards are a further signal of commercial potential. Given that SBIR is a highly competitive program, the acquisition of related SBIR awards also suggests that a project is moving along the development path toward commercialization, not least because com- mercialization potential has become a significant component in the decision to make an SBIR award. 43.5 percent of respondents indicated that they had received at least one additional related SBIR award.12 B. SBIR is in broad alignment with the needs of the DoD agencies and components. 1. DoD’s SBIR program has contributed to significant enhancements of its mission capabilities. A central mission of the DoD SBIR program is to use the inventiveness of small companies to solve DoD’s technical problems, and to develop new technologies that can be applied to the weapons and logistics systems that are eventually used by the Armed Forces.13 General Accounting Office, Small Business Innoation Research Program Shows Success But Can Be Strengthened, RCED–92–32, Washington, DC: U.S. General Accounting Office, 1992. 9 Derived from NRC Phase II Survey, Question 23. See also Table App-A-37. 10 Ibid. 11 NRC Phase II Survey, Question 23. See also Table App-A-37. Similar to sales, the amount of venture funding is skewed with only eight projects reporting $5 million or more in venture funding. 12 See Table 4-12. 13 Interviews with SBIR program managers.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE BOX 2-1 ArmorWorks, Inc.—Body Armor in Iraq Technologies developed from SBIR-funded research efforts were used in the design of Small Arms Protective Inserts (SAPI) Body Armor Plates used in the Interceptor Vest currently being worn by U.S. service men and women in the Middle East. ArmorWorks has been awarded more than $50 million in contracts from the Army and Marines to produce SAPI plates for body armor, making them a leading producer for the U.S. military. To date, some 350,000 SAPI plates have been produced for the Department of Defense. ArmorWorks also manufactures vehicle armor. The company’s SBIR research contributed to the design of HMMWV and add-on armor kits for trucks currently in use in the Middle East. The vehicles armor produced by ArmorWorks has a num- ber of valuable features for the battlefield, including easy installation (requires no vehicle modification or special tools) and field configurability (contains simplified installation to allow for reconfiguration for specific missions). ArmorWorks recently received another $30 million contract from the U.S. Defense Logistics Agency to produce and deliver Kevlar inserts to protect U.S. troops in Iraq and Afghanistan against small arms fire. SOURCE: DoD SBIR Success Stories, . Improved mission capabilities in the context of DoD relate to main- taining technological dominance in battle space conditions, increased responsiveness to new, unexpected situations, such as responding to improvised explosive devices (IEDs), and reductions in the cost of operations and support systems. DoD SBIR program managers also speak favorably about the cre- ativity of the small- and medium-sized firms that comprise the SBIR community.14 2. DoD appears to be making a concerted effort to ensure that SBIR proj- ects are aligned with the needs of the weapons system managers. 15 The topic development process has since 1999 incorporated important elements specifically designed to align acquisitions needs and SBIR topics.16 14The agency’s Technical Point of Contacts (TPOCs) were surveyed by the NRC. See NRC Project Manager Survey in National Research Council, An Assessment of the SBIR Program at the National Aeronautics and Space Administration, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2009. 15A number of initiatives in this direction are described in Chapter 5: The Phase III Challenge. 16 See Section 6.2.

OCR for page 23
7 FINDINGS AND RECOMMENDATIONS Millions of Dollars Year FIGURE 2-1 Reported Phase III contract awards value 1999–2005. SOURCE: John Williams, Navy, April 7, 2005. 2-01 According to DoD managers, 60 percent of topics published in the SBIR solicitation are now sponsored or otherwise supported by acqui - sition agencies.17 Based on DoD data, identified Phase III contracts18 now total more than $450 million19 annually, and have been growing rapidly in recent years. 3. SBIR offers an unusual degree of execution year flexibility and a short 17The Navy finds that 80 percent of its topics are acquisition linked. Comments by Dr. Holland and Mr. Caccuitto at the NRC Phase III Symposium, June 2005. See National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit. 18 Many Phase III contracts are not captured effectively by the DD350 reporting system, which undercounts the number and size of Phase III contracts awarded, to a varied degree at different DoD components. See Chapter 5. In addition, commercialization occurring intraindustry, at a subcontract level, or in the commercial marketplace (and then perhaps finding its way to DoD) is not measured by this data source. The DD350 has been replaced by the Federal Procurement Data System—Next Generation (FPDS-NG) as the system tracking, and thus characterizing, all prime contract actions. This system is now standard for the federal government, with a small number of exceptions. 19 $450 million is based on the DD 350: however, the 2005 DoD Annual SBIR Report to SBA lists by contract $565 million in known Phase III contracts and that listing is considered to be incomplete. It includes only contracts known to the SBIR program managers. It should be kept in mind that these contracts are for direct sales to DoD or direct further DoD R&D funding. Sales and further R&D from DoD primes are not included. The NRC Phase II Survey and the DoD Commercialization data indicate the DoD sales and funding are actually less than half of the total commercialization from DoD SBIR.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE planning horizon, permitting the program to rapidly address urgent mission needs. SBIR-supported innovations have contributed to enhanced U.S. com- bat capabilities, and provided technological solutions to meet sudden, unexpected challenges to our military. Indeed, the high degree of flex- ibility characteristic of small firms means that SBIR has provided DoD with an increased number of suppliers capable of quickly responding on short notice to unexpected battlefield situations.20 For example, the Navy has taken advantage of this flexibility to is- sue a “quick response IED topic” in 2004, and made 38 Phase I awards within 5 months of topic development. These developed into 18 Phase II awards, and results from these will be available in 2006–2007. The first prototypes were expected in Iraq in the Fall of 2006. By contrast, most RTD&E accounts require considerable forward planning. 4. SBIR increases the number of potential suppliers for new technolo- gies, and also creates new opportunities for these firms to partner together in new undertakings. Used effectively, SBIR can act as a low-risk, low-cost technology probe and a search tool for finding new technology suppliers. It has helped DoD personnel learn about new technologies, new applications, and a new set of high-tech firms with whom they would not otherwise have contact.21 The laws governing the SBIR program permit the use of sole source procedures when federal agencies acquire technologies developed with SBIR funding. This allows substantially faster acquisition than through standard channels and acts as a powerful incentive for SBIR firms and their partners. 5. The quality of SBIR-funded research is broadly comparable to that of other non-SBIR research according to the NRC Project Manager Survey. 20 See, for example, the case of ArmorWorks, Inc., in Box 2-1. 21 Firms express this outcome as follows: Had it not been for SBIR, their business with DoD ser- vices or agencies would not have developed. Services likely would have stayed with their pre-existing sources of supply. Program managers are normally too busy administering multiple contracts to search out or respond attentively to new sources of technology. Their propensity is to hire a contractor to solve problems rather than seek out the most technologically innovative performer. The SBIR program requires that program managers become involved with small firms, to look at technical options, and to allow for increased competition in the selection of R&D performers.

OCR for page 23
9 FINDINGS AND RECOMMENDATIONS In the NRC Project Manager Survey, 53 percent of Technical Points of Contact (TPOC) respondents indicated that the specific SBIR project identified in the survey produced results that were useful to them and that they had followed up on this work with other research. SBIR proj- ects are normally part of a wider portfolio of research responsibilities handled by TPOCs.22 The NRC Project Manager Survey also indicated that the quality of SBIR funded research is comparable to non-SBIR research they man- age. Normalized survey scores indicate that the quality of SBIR re - search is equivalent to that of other research at DoD.23 6. The Department has devoted considerable recent effort to strengthen the critical connection between SBIR and the acquisition programs through Phase III. While this focus is to be commended, the Phase III process can be considerably widened and improved. A striking aspect of SBIR Phase III at DoD is the extraordinarily uneven character of outcomes and activities between services, and between components within services. Notably, Phase III transitions at PEO SUBS account for approximately 86 percent of all Navy Phase III contracts, and Navy in turn accounts for about 70 percent of all DoD Phase III contracts, as captured by the DD350 forms completed by contracting officers.24 This skew partly results from the additional effort made by Navy to ensure that DD350 forms are completed and accurately reflect SBIR contributions, which in turn reflects different views of the impor- tance of supporting SBIR, as evidenced by the amount of resources, staff, and funding that services and components allocate for program support. At some components, such as Navy, senior management recognizes the potential value of SBIR and has supported extensive efforts to build effective bridges between SBIR and the acquisition programs. At other components, efforts have been less well supported, and on the basis 22 See the related discussion in Section 4.3.1.2. 23The scores were normalized scores by removing the outliers in the top and bottom 5 percent of scores. Statistical procedures often assume that the variables are normally distributed. A significant violation of the assumption of normality can seriously increase the chances of a Type I (overestima - tion) or Type II (underestimation) error. Nonnormality can occur in the presence of outliers (scores that are extreme relative to the rest of the sample). Removing the outliers can improve the normality of the distribution. See C. M. Judd and G. H. McClelland, Data Analysis: A Model-Comparison Ap- proach. San Diego, CA: Harcourt Brace Jovanovich, 1989. 24 See the discussion in Section 5.2, including Figure 5-1.

OCR for page 23
0 SBIR AT THE DEPARTMENT OF DEFENSE of the data presented in Chapter 5, they appear to be considerably less effective. 7. There is no effective and comprehensive tracking system within DoD to follow SBIR-funded technologies to their final outcome. The Company Commercialization Report is self-reported data, and must be updated only when a company applies for a new DoD SBIR. The agency’s DD350 reporting system may substantially undercount Phase III awards, as contracting officers must be specifically trained to capture this data correctly.25 The lack of a reliable and effective tracking system for SBIR awards that would identify follow-on funding sets back efforts to assess the impact of the program and to document its successes. C. SBIR awards made by DoD support small businesses in a number of important ways. 1. S BIR awards have had a substantial impact on participating companies. Company Creation. Just over 25 percent of companies responding to the NRC Firm Survey indicated that they were founded entirely or partly because of a prospective SBIR award.26 The Decision to Initiate Research. Only 13 percent of DoD project respondents thought that their project would “definitely” or “probably” have gone ahead without SBIR funding. Over two-thirds (about 70 per- cent) thought they definitely or probably would not have initiated the research; most of those who anticipated that their project would have gone ahead without the award acknowledged the likelihood of substan- tial delays without the award.27 Company Growth. Almost half (48 percent) of the respondents in- 25There are multiple limitations to current systems for tracking SBIR awards. The DD350 report - ing system can be used to extract some SBIR data; however, the system was not designed to gather SBIR award data, and is used differently by the services and agencies. For example the Navy reports Phase II Enhancements as Phase III, while the other services report such awards as Phase II. The Phase I and Phase II data in the DD350 does not match the SBIR budget or DoD Annual Report to SBA. The Annual Report to SBA, prepared by the DoD SBIR program managers, accounts for the budgeted SBIR funding. The SBIR program managers also have less knowledge of the Phase III awards, since these are made with funds that are not under control of the SBIR program. 26 See Table 4-15 (NRC Firm Survey, Question 1). Data reported in Table 4-15 are for firms with at least one DoD award. NRC Firm Survey results reported in Appendix B are for all agencies (DoD, NIH, NSF, DoE, and NASA). 27 See Figure 4-10 (NRC Phase II Survey, Question 13).

OCR for page 23
 FINDINGS AND RECOMMENDATIONS No Prior Awards 50 Percent of Projects Repor ting Firm SBIR of ThisPhase this Phase 45 Awards Prior to This Phase II 1 to 5 40 35 6 to 20 30 25 >20 20 15 10 5 Phase I Phase II 0 Phase I Phase II FIGURE 2-2 New winners at DoD. SOURCE: NRC Phase II Survey. dicated that more than half the growth experienced by their firm was directly attributable to SBIR.28 Partnering. SBIR funding is often used by small firms to gain access 2-02 to outside resources, especially academic consultants and, often, to seek company partners. 2. SBIR Awards Attract Participation by New Firms. FY2005 data from DoD show that 29 percent of Phase I awards went to firms that had not previously won a DoD Phase II SBIR award. DoD data indicate that a further 50 percent of awards went to compa - nies with five or fewer previous Phase II awards. Only 13 percent of awards went to companies with at least six Phase II awards.29 Data from the Naval Air Systems Command (NAVAIR) show that about half of all its Phase I contracts go to companies that have never won an SBIR award from NAVAIR before. In addition, about 40– 45 percent of Phase II contracts go to newly participating firms. 30 D. Multiple SBIR awards serve multiple objectives for firms and agencies. A small number of companies receive multiple awards. A few companies have 28 See Table 4-16. 29 Michael Caccuitto and Carol van Wyk, “Enhancing the Impact of Small Business Innovation Re - search (SBIR) Program: The Commercialization Pilot Program,” Presentation, September 27, 2006. 30 See Section 3.2.2.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE received more than 100 awards at DoD (over 20 years), and are higher in commercialization (on average).31 1. The myth of the linear innovation model: The program has often encountered criticism with regard to firms— sometimes labeled SBIR “mills”—that are described as winning large numbers of awards but that achieve lower levels of commercialization than firms with significantly fewer awards.32 Implicit in this view of multiple-award winners is the simple linear model of the program. In reality, multiple awards serve a variety of functions, as noted below. 2. Role of multiple awards. More than one award is often required to de- velop a technology or firm capability, quite apart from the commercializa- tion of a complete product. These needs reflect the complex and differing characteristics of firms. Diversity of firm objectives. Reflecting the diversity of the program’s objectives and of the participants, some firms approach the SBIR award process at different stages of development and with different objec- tives. Some firms are developing technology concepts; some firms see their vocation as contract research organizations; others actively seek to develop commercial products, either for public agencies or for the marketplace.33 Diversity in firm strategy. For example, investigator-led firms, lim- ited in size and focused on a single concept may seek multiple awards as they advance research on a promising technology.34 For firms that 31 It is important to keep in mind the difficulties in tracking companies over time. Companies regu - larly change names, locations, even employer identification numbers, which makes tracking them across time within the DoD awards database difficult. 32 See the 1992 GAO report, U.S. General Accounting Office, Small Business Innoation Research Program Shows Success But Can Be Strengthened, RCED–92–32, op. cit. That report focused on firms that had received twenty or more awards and on a single program metric, that is, commercial - ization. Congressional legislation later fixed fifteen awards in a five-year period as the level where efforts to commercialize should be taken into account. 33 See Reid Cramer, “Patterns of Firm Participation in the Small Business Innovation Research Pro - gram in the Southwestern and Mountain States,” in National Research Council, The Small Business Innoation Research Program: An Assessment of the Department of Defense Fast Track Initiatie, op. cit., p. 151. The author describes the incremental nature of technical advance, which sometimes necessitates several awards. See also John T. Scott, “An Assessment of the Small Business Innovation Research Program in New England: Fast Track Compared with Non-Fast Track,” in National Re - search Council, The Small Business Innoation Research Program: An Assessment of the Department of Defense Fast Track Initiatie, op. cit., p. 109 for a discussion of Foster-Miller, Inc. 34 Ibid. The mirror image of this approach is the program manager who makes several awards for similar technologies among different companies. In fact, it is not uncommon to have multiple award - ees on the same Phase I topic. For an example, see the Navy’s SBIR Web site selections page for their FY-06.1 awardees at . This page not only shows

OCR for page 23
 FINDINGS AND RECOMMENDATIONS carry out research as a core activity, success is often measured in mul- tiple contract awards. 3. Addressing agency missions. Some firms, mainly at DoD, have won large numbers of awards over the life of the program. Yet, even with many awards, there is nothing intrinsically wrong with a process that makes high-quality research available to the department at relatively low cost. Some of this research is intrinsically noncommercial, but may have considerable value.35 Identifying dead ends. Inexpensive exploration of new technological approaches can be valuable, particularly if they limit expenditure on technological dead ends. For research oriented firms, the key issue is the quality of the research and its alignment with service and agency needs.36 Each of the seven most frequent winners, who have received over 100 Phase II awards since the program inception, has a large num- ber of researchers who submit proposals. The high number of quality proposals can produce a high number of awards. Some successful ap- plicants use spin-off firms to commercialize the results of their SBIR awards. Providing solutions. In some cases, firms respond to an agency solici- tation and “solve” the problem, provide the needed data, or propose a solution that can then be adopted by the agency with no further “com- mercialization” revenues for the firm.37 several awardees for each topic, but if you click on the “Details” link, you can see the differences in companies’ approaches to the topics. 35To secure additional awards, a small company has to submit its proposal for follow-on research through the regular review process. These awards are relatively small in amount—the normal Phase I and Phase II awards would total $850,000. As a point of comparison, the top three U.S. prime contrac- tors in 2007 garnered over $20.5 billion in defense revenues. See Washington Technology, “Top 100 Federal Prime Contractors: 2004,” May 14, 2007. 36 For example, Foster-Miller, the most frequent SBIR award winner, provided armor for Humvees and aircraft and developed robots for use in Iraq to identify roadside improvised explosive devices. (This company has since been acquired, making it ineligible to participate in SBIR. See Financial Times, “Qinetiq set to make its first US acquisition,” September 8, 2004.) Their LAST® Armor has had sales in excess of $170 million. 37There are cases where a small business successfully completes the requirements and objectives of a Phase II contract, meeting the needs of the customer (e.g., by delivering specialized software or hardware), without gaining additional commercialization revenues. For example, Aptima, Inc., designed an instructional system to improve boat-handling safety by teaching the use of strategies that mitigate shock during challenging wave conditions. A secondary goal was to demonstrate how an innovative learning environment could establish robust skill levels while compressing learning time. Phase I of the project developed a training module, and in Phase II, instructional material, including computer animation, videos, images, and interviews were developed. The concept and the supporting materials were adopted as part of the introductory courses for Special Operations helmsmen with the goal of reducing injuries and increasing mission effectiveness.

OCR for page 23
 FINDINGS AND RECOMMENDATIONS bility, etc.) from these companies for all their previous SBIR awards, including those at other agencies. It also captures information on firm size and growth since entering the SBIR program, as well as the percent of annual revenue derived from SBIR awards. These historical results of prior awards are then used in proposal evaluation. Non-DoD agencies should consider adapting both this approach and the DoD technology and contributing to the DoD database. This would provide a unified track- ing system. Adaptations could be made to track additional data for specific agencies, but this would provide a cost-effective approach to enhance data collection on award outcomes. Multiple agencies. Gap-reduction strategies. The agencies have, to different de- grees, recognized the importance of reducing funding gaps. While details vary, best practice would involve development of a formal gap-reduction strategy with multiple components covering application, selection, contract negotiation, the Phase I–Phase II gap, and support after Phase II. DoD Phase II Enhancement (and NSF Phase IIB). The matching-fund approach adopted by NSF for Phase IIB and DoD for Phase II Enhancement might be explored at other agencies. The NSF matching requirement represents an important tool for helping companies to enter Phase III at nonprocurement agencies. The DoD funding match by acquisition programs provides a transition link into Phase III contracts with the agency. DoD-Navy. Technology Assistance Program. The Navy has developed the most comprehensive suite of support mechanisms for companies entering Phase III, and has also developed new tools for tracking Phase III outcomes. These are important initiatives, and other components and agencies should consider them carefully. recent reviews represent a positive effort to connect data and analysis to practice.56 6. Phase III results remain uneven across the services and among com- ponents within the services. This suggests that other components and other services could improve the performance of their programs. Interviews suggest that some elements of the department have not fully integrated SBIR within their own program missions and have not 56 See also a recent RAND Report requested by DoD: Bruce Held, Thomas Edison, Shari Lawrence Pfleeger, Philip S. Anton, and John Clancy, Ealuation Recommendations for Improement of the Department of Defense Small Business Innoation Research (SBIR) Program, Santa Monica, CA: RAND Corporation, 2006.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE provided SBIR with the resources and management attention needed to maximize its effectiveness.57 The 2005 NRC Symposium on SBIR Phase III contributed to the awareness of the SBIR program’s potential, the challenge that promis- ing products face in the Phase III transition, and the need for additional efforts to “team” across agencies, with SBIR program managers, Pro - gram Executive Officers, and prime contractors.58 H. Prime contractors are taking a positive approach towards the SBIR program. 1. Increased interest in SBIR. As the program has grown in size and perfor- mance, it has garnered greater attention from the DoD upper management and, importantly, the prime contractors. There is considerable evidence that prime contractor interest in—and engagement with—the SBIR program has been growing rapidly in recent years (see Chapter 6). Concretely, this is reflected in growing contractual linkages.59 2. Interest has been followed by action. This increased focus on the pro- spective contributions of the SBIR program by the prime contractors ap - pears to represent a significant positive endorsement of the contributions of the program. Steps taken by the prime contractors to integrate SBIR within their 57 Some DoD program managers see the SBIR program as a “tax,” that is an unwarranted allocation of funds to small business, one that hinders effective R&D program management and one that is time consuming to manage and without funds to cover the cost of management. For a more positive view of the program, see the discussion of the Navy program in the Findings section. The recommendations address program perception and incentives. 58 Following the National Academies meeting on the SBIR commercialization challenge, the Senate Small Business & Entrepreneurship Committee proposed legislation that established a Commercial - ization Pilot Program. See Section 252 of the 2006 National Defense Authorization Act. The bill was passed in bipartisan spirit by the Senate Committee on Small Business & Entrepreneurship (SBE) under the leadership of the Committee Chair, Olympia Snowe (R-ME) and Ranking Member, John Kerry (D-MA). Further reflecting the growing appreciation of the program’s role and the increased focus on Phase III, Dr. Finley has described the SBIR program as a means of accelerating innovation and putting better equipment into the hands of the war-fighter. See remarks by Deputy Under Secre - tary of Defense for Acquisition and Technology Finley at the Small Business Technology Coalition Conference on September 27, 2006. 59 Raytheon, for example, estimates that the value of technology leveraged through SBIR jumped from $3.8 million in 2004 to $11.6 million in 2005, and looks set to grow as rapidly in 2006. Raytheon is involved with 36 Phase I projects, 17 Phase II projects, 4 Phase III projects (with three more in the works), and has been a subcontractor on other projects. See Lani Loell, SBIR Program Manager Raytheon Integrated Defense Systems, Presentation to SBTC SBIR in Transition Conference, Sep - tember 27, 2006, Washington, DC.

OCR for page 23
 FINDINGS AND RECOMMENDATIONS strategic roadmaps reveal that they see the program contributing to technological innovations that further the Defense mission.60 At the Academies’ Phase III conference, representatives of prime con - tractors stated that there was already a substantial amount of prime involvement with the SBIR program. Moreover, several of the primes affirmed that they had made significant efforts to increase their levels of involvement. For example, Boeing had recently decided to increase its emphasis on SBIR. Similarly, at Raytheon, some divisions (e.g., Integrated Defense Sys- tems) had formal working arrangements with SBIR for several years. NRC STUDY RECOMMENDATIONS As noted in the Findings section above, the Department of Defense has an ef- fective SBIR program. The recommended improvements listed below should enable the DoD SBIR managers to address the four mandated congressional goals in a more efficient and effective manner. A. Improve the Phase III transition. DoD should continue to expand its work on improving the Phase III transition (the transition from SBIR-funded Phase I and Phase II research to commercialization—especially testing and evaluation funded by other DoD sources). It is important to recognize that the transition of new technologies is a complex process requiring teaming across areas of responsibility, additional resources, and often coping with some ele - ment of additional risk.61Areas for possible action include: 1. Aligning incentives. For the SBIR program to achieve its full potential, better incentives are required. Expansion of positive incentives for program officers to utilize the SBIR program for their own research needs, beyond the current requirements for their involvement in topic development. Management needs to improve incentives so that acquisitions officers perceive reduced risks and enhanced benefits from participating in the program. 2. Increasing resources. SBIR managers need greater resources to “match” program funds to encourage uptake. In addition to increased Phase III 60 See Section 5.4.4 and the discussion in Chapter 6. 61 See the discussion of these problems and potential remedies in the section on Phase III Transition in this volume. See also National Research Council, SBIR and the Phase III Challenge of Commer- cialization, op. cit.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE SBIR funding, linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition. 3. Developing an evaluation culture. Agency and service managers should have effective data collection and analysis as performance metrics. 4. Involving acquisition officers. Active participation by acquisition of- ficers is key to successful Phase III transitions. Acquisition officers control the funding, and their involvement is im - portant for successful commercialization of SBIR technologies. A cul - tural shift in program participation and use seems to have occurred at Navy once Program Executive Offices (PEO) became active champi- ons of SBIR involvement in acquisitions. Senior management support and encouragement, better information flows, improved PEO education about SBIR, and additional incen- tives for PEOs to use SBIR are all elements of an effective overall program. 5. Integrating with roadmaps. The long technology development and ac- quisition cycle for major weapons and logistics systems means that effec - tive Phase III transition requires early integration of SBIR topics and firms into the planning process.62 6. Linking SBIR with other programs. Linkages with other programs (e.g., ManTech) might be enhanced to facilitate the Phase III transition. 7. Improving outreach and matchmaking. There are significant barriers to the flow of information among SBIR firms, prime contractors, and acquisi- tions offices. Effective transition requires that these barriers be overcome, most likely through implementation of a range of activities, including improved electronic communications methods and matchmaking services like the Navy Opportunity Forum. In particular, efforts should be made, as appropriate, to align the SBIR program with the needs of the prime contractors responsible for the development of major systems. 8. Connecting with the primes. The growing interest among prime contrac- 62 In the case of the semiconductor industry, the industry perceived early on that problems of coor- dination could arise with a complex technology, multiple participants, and many ways of proceeding. This realization led to cooperative efforts led by SEMATECH to develop a technology roadmap setting out the relationships among science, technology, and applications as a point of reference for the researchers, technologists, project managers, suppliers, and users involved in and affected by the consortium’s work. As a general approach, roadmaps can advance similar coordination functions in other industry partnerships and in this way contribute to more efficient and more cooperative research. See National Research Council, Goernment-Industry Partnerships for the Deelopment of New Technologies: Summary Report, Charles W. Wessner, ed., Washington, DC: The National Academies Press, 2003.

OCR for page 23
 FINDINGS AND RECOMMENDATIONS tors of the SBIR program’s outputs and opportunities for partnering with SBIR companies should be encouraged. Consideration should be given to performance incentives to further encourage development of SBIR sup- ported technologies. 9. Assessing and expanding commercialization programs. Commercial- ization programs that provide training, counseling, and networking op- portunities should be assessed and, as appropriate, expanded. B. DoD should take immediate steps to enhance the perception of the SBIR program’s potential and accomplishments, promoting SBIR as an opportunity. A key element in the program’s operation is the attitude taken towards the program by the different levels of management in the Defense research and development community. Where SBIR is seen as an unwarranted intrusion on program man- agement, a “tax” on R&D resources, it is less likely to be effectively aligned with service needs and less likely to have the resources to develop and ultimately insert the results of successful Phase II tech- nologies in weapons and logistics systems and other programs. When the program is seen as an effective tool to engage the ingenuity of small companies in support of the Defense mission, with shorter lead times and more flexibility, it is much more likely to have its results adopted and incorporated. In short, there is an element of circularity in developing measures to enhance program effectiveness and management’s guidance and re- wards for those managers who use the program effectively. Providing the resources and incentives for managers to see opportunity rather than obligation may well enhance program effectiveness. C. DoD should substantially strengthen and expand its evaluation efforts in order to further develop a program culture that is driven by outcomes, data, and internal and external evaluations. Efforts to identify outcomes should be improved, and evaluations63 should 63 For example, each SBIR award has a DoD technical monitor who serves as the contracting offi - cer’s technical representative (COTR). This individual monitors the contractor’s performance during Phase I and recommends or issues the invitation for Phase II, monitors performance of Phase II, and receives the contractor’s Phase I and II reports. Despite the importance of the TPOC’s role, there is no systematic attempt to use the technical monitor to evaluate the quality of the SBIR efforts and to facilitate Phase III. Since these individuals are not funded separately for SBIR, dissemination of SBIR outcomes and assistance in transition is a function of individual initiative and competing responsibili - ties. DoD should consider training, funding, and making better use of this valuable asset.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE be connected much more directly to program management. More attention should be devoted to the role and contributions of the Contracting Officer’s Technical Representative (COTR). It is important that DoD create the capability to use outcomes data to help assess best practice. New mechanisms need to be developed that allow for the efficient design, implementation, and subsequent assessment of pilot programs. Efforts should be increased to make sure that appropriate metrics and benchmarks are adopted and implemented by all units, components, and Services. D. DoD should encourage and support the development of a results-oriented SBIR program with a focused evaluation culture. 1. Effective oversight requires additional staff and funding. Effective management of a data-driven SBIR program requires the regular collection of higher-quality data and systematic assessment. Cur- rently, sufficient resources are not available for these functions. Ad- ditional funding should be provided for program management and assessment. This funding should also be used to provide management oversight, including site visits, improved data collection and analysis, regular reporting, program review, and systematic third-party assessments. 2. To help foster an active evaluation culture, DoD should consider: Preparing an expanded annual SBIR program report. DoD should prepare an annual SBIR program report, which gathers all relevant data about awards, outcomes, program activities, and management initia- tives. In particular, the Department should publish detailed data annu- ally about Phase III take-up at each service, and at each component within each service, as well as providing information about program initiatives. Commissioning regular assessments. SBIR programs at both DoD and the individual DoD components should seek to enhance a data- driven management approach, with regular assessment supporting policy development and program management. Instituting systematic and objective, outside review. The internal assessment program should be supported and supplemented by sys- tematic, objective outside review and evaluation, as envisaged in the reauthorization legislation. Convening an advisory board. DoD should consider development of a formal advisory board, which would receive the annual program

OCR for page 23
7 FINDINGS AND RECOMMENDATIONS report and provide its own supplementary review of the report and management practices on an annual basis to senior DoD officials in charge of the SBIR program, or possibly to a subpanel of the Defense Sciences Board. 3. DoD should consider greater internal review and adoption of best practices. Such an assessment would identify best practices within DoD and develop mechanisms for encouraging other components to implement these prac - tices within their SBIR programs. One important example of best practice might be one focused on the Phase III transition at the Navy (see Box 2-3). Additional research should be undertaken to address, inter alia, three questions related to Navy practice. What unique factors make the Navy SUBS program successful? What role is played by initiatives at the service level in supporting the SUBS program? What elements of that success can best be transferred elsewhere in DoD as best practices to be followed? What changes will be needed to make those transfers successful? E. DoD should encourage and support pilot programs that evaluate new tools for improving the program’s overall performance. 1. Innovation through pilot programs. Making changes initially through pilot programs allows DoD to alter selected areas on a provisional basis; a uniform approach is unlikely to work well for all components of a program that funds highly diverse projects with very different capital requirements and very different product development cycles. 2. Some possible pilot projects include: Small Phase III awards. These could be a key to bridging the financ- ing ‘Valley of Death’ that many firms face in converting research to innovation to products.64 NASA for example sometimes provides a small Phase III award—perhaps enough money to fly a demonstration payload—for a technology not ready for a full Phase III. These might 64 See Lewis Branscomb and Philip Auerswald, Between Inention and Innoation: An Analysis of Funding for Early-Stage Technology Deelopment, NIST GCR 02-841, Gaithersburg, MD: National Institute of Standards and Technology, Prepared for the Economic Assessment Office, Advanced Technology Program, November 2002. See also National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.

OCR for page 23
 SBIR AT THE DEPARTMENT OF DEFENSE BOX 2-3 Lessons from the Navy Model Many of the issues we identify with regard to the SBIR program at the Department of Defense have been addressed, with considerable success, by the Navy SBIR program. Keeping in mind the appropriate caveats concerning different agency needs, operating conditions, and cultural traditions, a number of aspects of the Navy program address these concerns. Key features of the Navy program include: Positive acceptance of the program. Navy PEO’s and program managers increas- ingly appear to see the SBIR program as a useful tool in meeting mission objectives, as acquisition staff are drawn increasingly into topic development and SBIR project management. Top management focus. One reason for the positive perception of the program’s utility is that the Navy provides significant management attention, particularly at the program executive officer (PEO) level, to the integration of SBIR into technology development to meet program needs. Strong leadership from the Navy hierarchy emphasizes the potential of SBIR for Navy missions. Administrative funding and activities. The Navy provides substantial additional fund- ing, now on the order of $20 million per annum, to operate the program. These funds meet a variety of needs ranging from additional professional staff support, funding for the Technology Assistance Program, and resources for the Navy Opportunity Forum that helps match SBIR companies with potential customers. be combined with milestones or gateways to additional rounds of Phase II funding. Unbundling larger contracts. Organizing larger contracts into smaller components would tend to open more Phase III opportunities for SBIR firms.65 Redefining testing and evaluation within SBIR. DoD could pilot adoption of a wider view of RDT&E, so that SBIR projects could qualify for limited testing and evaluation funding. That in turn would help fund improvements in readiness levels. “Spring loading” Phase III, by putting in place Phase II milestones that could help to trigger initial Phase III funding. This could possibly 65The recent focus for DoD research, development, and engineering centers has been to take many small support contracts, which were then done by small business and bundle them into a large solici - tation, which can only be won by a firm which has significant resources. Bundling limits options for follow-on engineering support by SBIR firms. An alternative to unbundling is to provide incentives within large omnibus support contracts that make subcontract awards to SBIR firms.

OCR for page 23
9 FINDINGS AND RECOMMENDATIONS Emphasis on Phase III funding and process. Navy PEO’s have embraced the chal- lenge of maturing innovative technologies (including SBIR products), which requires serial funding for the many testing, evaluation and demonstration steps that precede acquisition. Demonstration effects and program integration. The strong Phase III take-up re- cently demonstrated at Navy suggests that acceptance of the program as a valid and useful component in the Navy’s overall technology development strategy creates a vir- tuous cycle. Successfully transitioned technologies such as the SAVI logistics tracking system, the DSR sonar, the ACR’s “Silver Fox” reconnaissance UAV, and cost-saving diagnostic technologies provide powerful demonstration effects, underscoring the po- tential contributions of the program to meet a broad range of Navy needs. Documented achievement. One of the distinctive features of the programs at Navy is that it successfully documents its accomplishments.a Data from DD350 reports shows that Navy’s Phase III contracts grew from $50 million in 2000 to $350 million in 2005. While these growing achievements may in part reflect unique or superior record keeping as compared with other services, this in itself reflects successful adoption of a data-driven assessment culture. The data gathered provides superior information and improved understanding of the operation and potential of the program. aOne factor contributing to the Navy’s assessment culture is the tradition of research excel- lence by the Office of Naval Research, one of the nation’s preeminent federal research funding agencies. occur in the context of larger, staged, Phase II awards in which ad- ditional stages fund more Demonstration and T&E, where non-SBIR funds or resources are leveraged. 3. Evaluating pilots. DoD should develop a formal mechanism for de- signing, implementing, and evaluating pilot programs. Pilot programs allow agencies to investigate program improvements at lower risk and potentially lower cost. Effective pilot programs require rigorous design and evaluation, clear metrics for success, and the necessary resources and internal support. 4. A flexible approach is required. In some cases, pilot programs may require waivers from SBA for activities not otherwise permitted under the SBA guidelines.66 SBA should be encouraged to take a highly flexible view of all agency proposals for pilot programs. 66 In the past, SBA has shown commendable flexibility in allowing agencies to deviate from stan - dard award sizes to accommodate the needs of the technology in question and to devise program innovations, such as the NSF Phase IIB, to provide incentives for commercialization.

OCR for page 23
0 SBIR AT THE DEPARTMENT OF DEFENSE F. Provide additional management resources. To carry out the measures recommended above to improve program utilization, management, and evalu- ation, the program should be provided with additional funds for management and evaluation. 1. Effective oversight relies on appropriate funding.67 A data-driven program requires high quality data and systematic assessment. As noted above, suf - ficient resources are not currently available for these functions. 2. Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other nec- essary management activities. 3. In considering how to provide additional funds for management and evalu- ation, there are three ways that this might be done: Additional funds might be allocated internally, within the existing budgets of the services and agencies, as the Navy has done. Funds might be drawn from the existing set-aside for the program to carry out these activities. The set-aside for the program, currently at 2.5 percent of external research budgets, might be marginally increased, with the goal of pro - viding management resources necessary to maximize the program’s return to the nation.68 G. DoD should take steps to increase the participation and success rates of woman- and minority-owned firms in the SBIR program. 67According to recent OECD analysis, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent for small programs and 1 percent for large-scale programs. See “Evaluation of SME Policies and Programs: Draft OECD Handbook,” OECD Handbook, CFE/SME(2006)17. 68 Each of these options has its advantages and disadvantages. For the most part, the departments, institutes, and agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legisla - tion does not permit this and would have to be modified; the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an over- all increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year, or roughly double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot. See also the National Research Council, SBIR and the Phase III Challenge of Commercialization, op. cit.

OCR for page 23
 FINDINGS AND RECOMMENDATIONS 1. Encourage participation. Develop targeted outreach to improve the par- ticipation rates of woman- and minority-owned firms, and strategies to improve their success rates. These outreach efforts and other strategies should be based on causal factors determined by analysis of past proposals and feedback from the affected groups.69 2. Encourage emerging talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, especially in the biomedical sciences. This means that many of the woman and mi- nority scientists and engineers with the advanced degrees usually neces - sary to compete effectively in the SBIR program are relatively young and may not yet have arrived at the point in their careers where they own their own companies. However, they may well be ready to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. Over time, this talent pool could become a promising source of SBIR participants. 3. Improve data collection and analysis. The Committee also strongly encourages the agencies to gather and report the data that would track woman and minority firms as well as principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity. 69This recommendation should not be interpreted as lowering the bar for the acceptance of pro - posals from woman- and minority-owned companies. Rather it should be seen as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups.