Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.

OCR for page 19
2 Findings and Recommendations I. NRC STUDY FINDINGS A. The NIH SBIR program is making significant progress in achieving the congressional goals for the program. The SBIR program is sound in concept and effective in practice at NIH. With the programmatic changes recom- mended here, the SBIR program should be even more effective in achieving its legislative goals.1 1. Overall, the program has made significant progress in achieving its congressional objectives by: Stimulating technological innovation; Using small business to meet federal research and development needs; Fostering and encourage participation by minority and disadvantaged persons in technological innovation; and Increasing private sector commercialization of innovations derived from federal research and development. B. The NIH SBIR program is focused on commercialization and has seen 1 Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992, (PL 102-564) Congress expanded the purposes to “emphasize the program’s goal of increasing private sector commercialization developed through Federal research and development and to improve the Federal government’s dissemination of information concerning small business innovation, particu- larly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.” 

OCR for page 19
0 SBIR AT THE NATIONAL INSTITUTES OF HEALTH meaningful achievement. There are, nonetheless, opportunities for im- provement in commercialization. 1. A significant percentage of SBIR projects are commercialized to some degree. i. Reaching the market. NRC Phase II Survey data suggest that 40 percent2 of SBIR-funded projects reach the marketplace.3 Over time, NIH data suggests that this figure will rise significantly; subsequent assessment is required to capture this trend. ii. Revenue skew. The survey data also show that a much smaller num- ber (7.9 percent of NRC Phase II Survey respondents) of projects generate more than $5 million in revenues.4 This type of “skew” or concentration—in which a majority of projects are at least modestly successful while a small proportion earns large revenues—is typical of early-stage finance.5 iii. Licensing revenue. In some cases, substantial licensing revenues have been generated on the basis of SBIR-funded projects.6 iv. Additional private investment. Some companies have received sub- stantial additional investment from the private sector, or have been 2 Forty point seven percent of NRC Phase II Survey respondents reported sales. The NIH Survey found that 30.3 percent of the projects surveyed reached the marketplace. National Institutes of Health, National Surey to Ealuate the NIH SBIR Program: Final Report, July 2003. 3 See Figure 4-1. 4 See Figure 4-2. One of the 496 projects recently surveyed by the NRC generated revenues of more than $50 million. Case studies identified other projects not included in the survey with similar results (e.g., Optiva, Martek). 5As with investments by angel investors or venture capitalists, SBIR awards result in highly con- centrated sales, with a few awards accounting for a very large share of the overall sales generated by the program. These are appropriate referent groups, though not an appropriate group for direct comparison, not least because SBIR awards often occur earlier in the technology development cycle than where venture funds normally invest. Nonetheless, returns on venture funding tend to show the same high skew that characterizes commercial returns on the SBIR awards. See John H. Cochrane, “The Risk and Return of Venture Capital,” Journal of Financial Economics, 75(1):3-52, 2005. Draw- ing on the VentureOne database Cochrane plots a histogram of net venture capital returns on invest- ments that “shows an extraordinary skewness of returns. Most returns are modest, but there is a long right tail of extraordinary good returns. 15 percent of the firms that go public or are acquired give a return greater than 1,000 percent! It is also interesting how many modest returns there are. About 15 percent of returns are less than 0, and 35 percent are less than 100 percent. An IPO or acquisition is not a guarantee of a huge return. In fact, the modal or ‘most probable’ outcome is about a 25 percent return.” See also Paul A. Gompers and Josh Lerner, “Risk and Reward in Private Equity Investments: The Challenge of Performance Assessment,” Journal of Priate Equity, 1(Winter 1977):5-12. Steven D. Carden and Olive Darragh, “A Halo for Angel Investors,” The McKinsey Quarterly, 1, 2004 also show a similar skew in the distribution of returns for venture capital portfolios. 6 See Table 4-7.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS bought by other companies, both of which indicate that the company has developed something of value.7 2. NIH has increased the significance of the commercialization compo- nent of applications over time. i. More efforts are now made to ensure that commercialization criteria are applied during Phase II selection. ii. NIH has developed several programs under the Technology Assis- tance Program aimed at helping awardees develop and implement ef- fective commercialization plans. Outside contractors have been hired to implement these programs.8 iii. However, because the focus on commercialization and the deploy- ment of assistance programs are recent, the impact of these efforts on commercialization is not yet clear, although initial results are encouraging, as participant firms have attracted $68 million in third party funding.9 3. SBIR-funded research projects enable small businesses to attract third-party interest. i. Venture funding. Third parties that identify substantial value in SBIR projects sometimes provided additional funding for the grantee com- pany. At least 50 of the 200 most frequent winners of NIH SBIR awards have received venture funding, and those investments totaled more than $1.5 billion (1992-2005).10 ii. Acquisition. In other cases, the technology developed had sufficient commercial potential that investors bought the grantee company out- right. For example, in 2000, Philips bought out SBIR recipient Optiva for a reported sum of more than $1 billion.11 iii. Multiple other sources. Many grantees have found additional funds from a wide range of sources, including angel funding. Fifty-eight percent of NRC Phase II Survey respondents attracted some additional investment (excluding further SBIR awards).12 7 See Table 4-11. 8 See Section 5.8.5.2—Commercialization Assistance Program. 9 See Section 5.8.5.2. 10 See Figure 4-7. Other analyses have put the number much higher. See U.S. General Account- ability Office, Small Business Innoation Research: Information on Awards Made by NIH and DoD in Fiscal Years 00 through 00, GAO 06-565, Washington, DC: U.S. Government Accountability Office, 2006. 11 See Box 4-3 in Chapter 4. 12 See Table 4-9.

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH C. The NIH SBIR program is operated in alignment with the agency’s mission13: awards are made for research that supports improved health within the United States. 1. SBIR funds projects that have a positive impact on public health. i. Effective mission alignment. All NIH awards appear to be selected primarily on the basis of their potential to advance knowledge and provide solutions in the field of health care and biomedicine. There is no evidence that NIH awards are made in fields outside those linked to the agency’s mission. ii. Positive impact on healthcare. SBIR awards have had a substantial impact on many aspects of health care. For example, SBIR awards played an important role in the development of a retractable non- stick needle that makes immunization safer, labor saving advances in the monitoring of epileptics, communication technologies for the disabled, disease specific tests, and improved infant formulas that are sold worldwide. SBIR awards have also helped develop tools that are used by researchers such as an SNP genotyping system, educational CDs and videos, as well as devices with large impacts on small populations—such as the SBIR-supported heart stent—SBIR awards have also helped develop devices with smaller impacts on very large populations, such as the Sonicare electric toothbrush, along with many other improvements in medical technology and practice. The impact of an SBIR project on public health is carefully considered during the selection process. Grantees and NIH staff note that impact effects are an important component in every application. In all the cases examined, NIH SBIR funded projects related to public health and biomedical science and technology. D. The SBIR program at NIH has provided significant support for small busi- ness, frequently acting as the impetus for projects and firm creation. The NRC Phase II Survey and NRC Firm Survey show that the SBIR pro- gram has provided substantial benefits for participating small businesses in a number of different ways. Responses indicate that these benefits include: 1. Company creation. Just over 25 percent of companies indicated that they were founded entirely or partly because of an SBIR award;14 13 NIH’s mission “is science in pursuit of fundamental knowledge about the nature and behavior of living systems and the application of that knowledge to extend healthy life and reduce the burdens of illness and disability.” Access at . 14 See Table 4-20.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS 2. The project initiation decision. More than 50 percent of SBIR-funded projects reportedly would not have taken place without SBIR funding; 3. Alternative path development. Companies often use SBIR to fund alter- native development strategies, exploring technological options in parallel with other activities; 4. Partnering and networking. SBIR funding pays for outside resources, especially academic consultants and partners, thereby contributing to net- working effects and facilitating the transfer of university knowledge to the private sector; 5. Commercializing academic research. The partnering between academic institutions and private firms (noted above) and the role of academ- ics in founding firms contribute to the commercialization of university research.15 E. Support for minority- and woman-owned firms. Data from NIH raise con- cerns about the shares of awards being made to woman- and minority-owned firms. 1. Awards, applications, and success rates have all declined for minorities, for both Phase I and Phase II (see Figures 2-1 and 2-2), while awards for woman-owned firms have not kept pace with the growth in female Ph.D. recipients in the life scientists. 2. Further research is required to determine whether the pool of potential applicants is not growing fast enough to keep pace with expanded SBIR funding, or whether there are other explanations for these trends. 3. From 2003-2006, average Phase II success rates (awards as a percent- age of applications) for minority-owned businesses are almost 10 per- centage points lower than those of firms that are neither woman- or minority-owned. F. NIH SBIR awards are open to new entrants. 1. High proportion of new entrants. The Phase I share of previous non- winners is quite large, ranging between just under 50 percent in 2000 and just above 35 percent in 2005.16 As the number of successful participants in the program rises, the proportion of new entrants may be diminishing. Still, the awards are widely distributed, with more than 1,300 companies receiving at least one Phase II award from 1992 to 2002. 2. Few frequent award winners. Another measure of openness is the rela- 15 See Table 4-21. 16 See Figure 3-5.

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH 16 14 Percent of all Awards 12 Woman-owned firms 10 Minority-owned firms 8 Linear (Woman-owned 6 firms) Linear (Minority-owned 4 firms) 2 0 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 19 19 19 19 19 19 19 19 20 20 20 20 20 20 20 Year FIGURE 2-1 Share of Phase II awards to woman- and minority-owned firms, 1992- 2006. NOTE: Following discussions with the NRC staff, the NIH made an effort to recalculate the data for woman and minority owners’ participation in the SBIR program. In Septem- ber 2007, the NIH provided corrected data, which is shown in Appendix A and in several figures in this report. However, apparent anomalies in the NIH data on the participation of women and minorities innew 2-1 2001-2002 could not be resolved by the time of publication of this report. SOURCE: National Institutes of Health. Awards as Percent of Applications 60 50 40 Woman-owned firms 30 Minority-owned firms Other small businesses 20 10 0 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06 19 19 19 19 19 19 19 19 20 20 20 20 20 20 20 Year FIGURE 2-2 Success rates for Phase II applications and awards to woman- and minority- owned firms, 1992-2006. NOTE: Following discussions with the NRC staff, the NIH made an effort to recalculate the data for woman and minority owners’ participation in the SBIR program. In Septem- ber 2007, the NIH provided corrected data,new is shown in Appendix A and in several fig 2-2 which figures in this report. However, apparent anomalies in the NIH data on the participation of women and minorities in 2001-2002 could not be resolved by the time of publication of this report. SOURCE: National Institutes of Health.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS tively low number of frequent award winners at NIH. Only five companies have been identified as receiving more than 20 Phase II awards between FY1992 and FY2005, and only three received 30 or more, with the maxi- mum being 34.17,18,19 3. Improving access. The SBIR program at NIH has also made efforts to improve access to the program for researchers outside the “high-award” states. The number of states receiving one or zero Phase II awards de- clined from 28 in 1995 to 16 in 2003. Similarly, the percentage of Phase II awards going to California fell from 22.8 percent to 13.6 percent in that time period (though the actual number of awards increased in light of the substantial increase in NIH funding during the period). G. Venture funding and SBIR. 1. Synergies. There can often be useful synergies between angel and venture capital investments and SBIR funding; each of these funding sources tends to select highly promising companies. i. Angel investment. Angel investors often find SBIR awards to be an effective mechanism to bring a company forward in its develop- ment to the point where risk is sufficiently diminished to justify investment.20 ii. Venture investment. Reflecting this synergy, initial NRC review in- dicates about 25 percent of the top 200 NIH Phase II award winners 17 See Table 3-6. 18 NIH has declined to provide company identification data on privacy grounds, so multiple win- ners are calculated by matching company names. This approach may understate the full distribution of multiple-award winners, even though additional cross-checks of the data were made to reduce the impact of these inaccuracies. The accuracy of these data could be improved by using EINs if they became available. 19The top 20 percent of winning companies together received 11.1 percent of awards. This is sig- nificantly lower than the Department of Defense. 20 See Figure 4-7. See the presentation “The Private Equity Continuum” by Steve Weiss, Executive Committee Chair of Coachella Valley Angel Network, at the Executive Seminar on Angel Funding, University of California at Riverside, December 8-9, 2006, Palm Springs, CA. In a personal commu- nication, Weiss points out the critical contributions of SBIR to the development of companies such as CardioPulmonics. The initial Phase I and II SBIR grants allowed the company to demonstrate the potential of their products in animal models of an intravascular oxygenator to treat acute lung infec- tions and thus attract angel investment and subsequently venture funding. Weiss cites this case as an example of how the public and private sectors can collaborate in bringing new technology to markets. Steve Weiss, Personal Communication, December 12, 2006.

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH (1992-2005) have acquired some venture funding in addition to the SBIR awards.21 2. Program change. During the first two decades of the program, some venture-backed companies participated in the program, receiving SBIR awards in conjunction with outside equity investments. During this lengthy period, the participation of venture funded firms was not an issue. In a 2002 directive, the Small Business Administration said that to be eli- gible for SBIR the small business concern should be “at least 51 percent owned and controlled by one or more individuals who are citizens of, or permanent resident aliens in, the United States, except in the case of a joint venture, where each entity to the venture must be 51 percent owned and controlled by one or more individuals who are citizens of, or permanent resident aliens in, the United States.”22 The effect of this directive has been to exclude companies in which VC firms have a controlling interest. 23 i. It is important to keep in mind that the innovation process often does not follow a crisp, linear path. Venture capital funds normally (but not always) seek to invest when a firm is sufficiently developed in terms of products to offer an attractive risk-reward ratio.24 Yet even firms benefiting from venture funding may well seek SBIR awards 21The GAO report on venture funding within the NIH and DoD SBIR programs used a somewhat different methodology to identify firms with VC funding. As a result of the approach adopted, no conclusions can be drawn from the study as to whether firms identified as VC-funded are in fact excluded from the SBIR program on ownership grounds. In addition, the number of VC-funded firms—reportedly 18 percent of all NIH firms receiving Phase II awards from 2001-2004—is con- siderably higher than suggested by preliminary NRC analysis. U.S. General Accountability Office, Small Business Innoation Research: Information on Awards made by NIH and DoD in Fiscal years 00-00, op. cit. 22Access the SBA’s 2002 SBIR Policy Directive, Section 3(y)(3) at . 23This new interpretation of “individuals” resulted in the denial by the SBA Office of Hearings and Appeals of an SBIR grant in 2003 to Cognetix, a Utah biotech company, because the company was backed by private investment firms in excess of 50 percent in the aggregate. Access this decision at . The ruling by the Administrative Law Judge stated that VC firms were not “individuals,” i.e., “natural persons,” and therefore SBIR agencies could not give SBIR grants to companies in which VC firms had a control- ling interest. The biotechnology and VC industries have been dismayed by this ruling, seeing it as a new interpretation of the VC-small business relationship by SBA. See, for example, testimony by Thomas Bigger of Paratek Pharmaceuticals before the U.S. Senate Small Business Committee, July 12, 2006. 24The last 10 years has seen a decline in venture investments in seed and early stage and a con- comitant shift away from higher-risk early-stage funding. See National Science Board, Science and Engineering Indicators 00, Arlington, VA: National Science Foundation, 2006. This decline is reportedly particularly acute in early-stage technology phases of biotechnology where the investment community has moved toward later-stage projects, with the consequence that early-stage projects have greater difficulty raising funds. See the testimony by Jonathan Cohen, founder and CEO of 20/20

OCR for page 19
7 FINDINGS AND RECOMMENDATIONS as a means of exploring a new concept, or simply as a means of capitalizing on existing research expertise and facilities to address a health-related need or, as one participant firm explained, to explore product-oriented processes not “amenable to review” by academics who review the NIH RO1 grants.25 ii. Some of the most successful NIH SBIR award winning firms—such as Martek—have, according to senior management, been successful only because they were able to attract substantial amounts of venture funding as well as SBIR awards.26 iii. Other participants in the program believe that companies benefiting from venture capital ownership are essentially not small businesses and should therefore not be entitled to access the small percentage of funds set aside for small businesses, i.e., the SBIR Program. They believe further that including venture-backed firms would decrease support for high-risk innovative research in favor of low-risk product development often favored by venture funds.27 3. Limits on venture funding. The ultimate impact of the 2004 SBA rul- ing remains uncertain. What is certain is that no empirical assessment of its impact was made before the ruling was implemented. At the same time, the claims made by proponents and opponents of the change appear overstated. i. Preliminary research indicates that approximately 25 percent of the NIH SBIR Phase II winners have received VC funding; that some of these are now graduates of the program (having grown too large or left for other reasons), and some are also not excluded by the ruling be- cause they are still less than 50 percent VC owned. Yet it is important to recognize that these companies may be disproportionately among GeneSystems, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” July 28, 2005. 25 See the statements by Ron Cohen, CEO of Acorda Technologies, and Carol Nacy, CEO of Se- quella Inc, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” July 28, 2005. Squella’s Dr. Nacy’s testimony captures the multiple sources of finance for the 17-person company (June 2005). They included—founder equity investments; angel investments; and multiple, competitive scientific research grants, including SBIR funding for diagnostics devices, vaccines, and drugs. SBIR funding was some $6.5 million out of a total of $18 million in company funding. Dr. Nacy argues that SBIR funding focuses on research to identify new products while venture funding is employed for product development. 26 See Box 4-4 in Chapter 4. 27 See the testimony by Jonathan Cohen, founder and CEO of 20/20 GeneSystems, at the House Science Committee Hearing on “Small Business Innovation Research: What is the Optimal Role of Venture Capital,” op. cit. In the same hearing Mr. Fredric Abramson, President and CEO of Alpha- Genics, Inc., argues that “any change that permits venture owned small business to compete for SBIR will jeopardize biotechnology innovation as we know it today.”

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH the companies—such as previous highly successful SBIR companies that were also VC funding recipients Invitrogen, MedImmune, and Martek—most likely to generate significant commercial returns.28 What is not known is how many companies are failing to apply to the program as a result of the ruling. ii. For firms seeking to capitalize on the progress made with SBIR awards, venture funding may be the only plausible source of funding at the levels required to take a product into the commercial mar- ketplace. Neither SBIR nor other programs at NIH are available to provide the average of $8 million per deal currently characterizing venture funding agreements.29 iii. For firms with venture funding, SBIR may allow the pursuit of high risk research or alternative path development that is not in the primary commercialization path, and hence is not budgeted for within the primary development path of the company.30 4. An empirical assessment. As noted above, the SBA ruling concerning eligibility alters the way the program operated during the period of this review (1992-2002), as it has, presumably, from the program’s origin. Anecdotal evidence and initial analysis indicate that a limited number of venture-backed companies have been participating in the program. To bet- ter understand the impact of the SBA exclusion of firms receiving venture funding (resulting in majority ownership), the NIH recently commissioned an empirical analysis by the National Academies. This is a further positive step towards an assessment culture and should provide data necessary to illuminate the ramifications of this ruling.31 28 For discussion of the factors affecting the returns to venture capital organizations, including incentive and information problems and the role venture funds have played in supporting a limited number of highly successful firms, see P. Gompers and J. Lerner, The Venture Capital Cycle, Cam- bridge: The MIT Press, 2000, Ch. 1. 29 See National Venture Capital Association, Money Tree Report, November, 2006. The mean venture capital deal size for the first three quarters of 2006 was $8.03 million. This trend has been accelerated by the growth of larger venture firms. See P. Gompers and J. Lerner, The Venture Capital Cycle, op. cit., Ch 1. 30 Firms that have used SBIR in this manner include Neurocrine and Illumina. The latter indicated in interviews that these alternative paths later become critical products that underpinned the success of the company. 31This research will address questions such as: which NIH SBIR participating companies have been or are likely to be excluded from the program as a result of the 2002 rule change on Venture Capital Company ownership?; and what is the likely impact of the 2002 ruling had it been applied during the 1992-2006 timeframe and what is its probable current impact? Key variables will include the pres- ence and amount of SBIR support, the receipt of venture capital funding or other outside funding, and output measures including those related to commercialization and knowledge generation.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS H. Stimulating technological innovation. The SBIR program at NIH is fulfill- ing its mission to support the transfer of knowledge into the marketplace. In the process, it is encouraging the general expansion of medical knowledge. The program supports innovation and knowledge transfer in several ways: 1. Patents and publications. SBIR companies have generated numerous patents and publications, the traditional measures of knowledge transfer activity. Thirty-four percent of projects surveyed by NRC generated at least one patent, and just over half resulted in at least one peer-reviewed article.32 2. Knowledge transfer from universities. The NRC Phase II Survey and NRC Firm Survey also suggest that SBIR awards are supporting the trans- fer of knowledge, firm creation, and partnerships between universities and the private sector: i. In more than 80 percent of responding companies with projects at NIH, at least one founder was previously an academic.33 ii. About 33 percent of founders were most recently employed as aca- demics before the creation of their company. iii. About 34 percent of NIH projects had university faculty as contractors on the project, 24 percent used universities as subcontractors, and 15 percent employed graduate students.34 3. Indirect paths. There is strong anecdotal evidence concerning beneficial “indirect path” effects—that projects provide investigators and research staff with knowledge that may later become relevant in a different con- text—often in another project or even another company. While these ef- fects are not directly measurable, discussion during interviews and case studies suggest they exist.35 I. The NIH SBIR program has not benefited from regular evaluation. 1. Prior to the congressional legislation authorizing this study, no sys- tematic, external program assessment had been undertaken at NIH. 2. A culture of assessment is now developing. Significant progress has 32 See Table 4-23. Without detailed identifying data on these patents and publications, it is not feasible to apply bibliometric and patent analysis techniques to assess the relative importance of these patents and publications. 33 See Table 4-21. 34 See Table 4-25. 35 For a discussion of the “indirect path” phenomenon with regard to the results of innovation awards, see Rosalie Ruegg, “Taking a Step Back: An Early Results Overview of Fifty ATP Awards,” in National Research Council, The Adanced Technology Program: Assessing Outcomes, Charles W. Wessner, ed., Washington, DC: National Academy Press, 2001.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS 3. Modest management engagement. In many cases, SBIR responsibilities are a small part of an Institute and Center (IC) manager’s much larger port- folio of projects, and reportedly Institute and Center senior management interest in SBIR is often modest. An absence of management engagement with the program can negatively impact perceptions of the program as well as the resources and staff devoted to its operation. 4. Limited benchmarking for success. The SBIR Program Coordinator’s office appears to have few formal operational benchmarks for program success, other than compliance—i.e., the full annual disbursement of award funding. This is also true for individual Institutes and Centers that disburse funds and operate the program. 5. Limited analytic capacity and utilization. Decisions that affect the char- acter of the program are made and implemented in the absence of data- based analysis, and without clear benchmarks for assessing the success or failure of a given initiative. The recent increase in the mean and median size of Phase I and Phase II awards provides a good example. NIH staff have offered a number of different justifications for the change, but no systematic analysis or review appears to have been made beforehand, and no post hoc assessment of the impact is currently underway. K. Selection concerns. While some interviewees and staff believed that the NIH peer-based selection process is generally equitable and procedurally fair, the selection process generated the most criticisms both internally and externally. Verifying the accuracy of these criticisms is inherently difficult. They are cited here because they were repeatedly raised in interviews and should be reviewed in turn by the management. Key criticisms included: 1. Limited commercial review. The commercial potential of projects is often assessed by academic scientists who may have little knowledge of the marketplace. 2. Conflicts of interest. Some applicants fear that both academic and nonacademic reviewers may have conflicts of interest with proposals. The challenge, of course, is to find reviewers who are knowledgeable but do not have competing interests. 3. Timeliness. Some believe that insufficient effort is made to ensure that the review process is completed as rapidly as possible. This is espe- cially important for small business applicants that need to move for- ward expeditiously to take advantage of a time-sensitive opportunity. 4. Resubmission. The opportunity to resubmit proposals is a major advantage of the NIH program, because it allows applicants to fix minor problems with their proposals and resubmit the applications. It is often cited by NIH staff in response to criticisms of the selection

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH process. While a very positive mechanism, it should be understood that resubmission can impose real costs on small firms in a commer- cial environment where delayed funding brings about inefficiencies and lost opportunities. A more timely, targeted response to review mechanism may be required. II. RECOMMENDATIONS The recommendations in this section are designed to improve the operation of the NIH SBIR program.38 It is important to keep in mind that the program is achieving its legislative goals. Meaningful commercialization is occurring and the awards made under the program are making valuable additions to biomedi- cal knowledge and developing products to apply that knowledge to the nation’s health. With the programmatic changes recommended here, the NIH SBIR pro- gram should be even more effective in achieving its legislative goals. A. The NIH should increase commercialization and evaluation efforts, im- prove data collection, expand outreach, especially for minorities and women, develop a culture of critical evaluation, obtain additional man- agement resources for these tasks, and encourage upper management attention to better exploit the program’s potential. 1. Flexibility. It is most important that the program retain the flexibility and experimentation that have characterized its recent management. The SBIR program is effective across the agencies because a “one-size fits all” ap- proach has not been imposed. 2. Evaluation. Much greater effort is required to evaluate current outcomes, collect relevant data, including with regard to participation of minority- and woman-owned firms, and document the impact of changes to the program. i. Significant improvement in data collection and assessment is needed. ii. Efforts to identify outcomes across a variety of metrics should be improved. iii. Regular internal and external evaluations should be undertaken to enable managers to assess program performance and the results of management initiatives. 3. Innovation. Efforts to initiate program innovation by NIH should be substantially strengthened and encouraged with due regard for best prac- 38The Committee’s recommendations below are drawn from analysis of the NRC survey data, review of program operations, and discussions with program participants.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS tice lessons from other programs. Pilot programs, possibly for individual Institutes and Centers are one mechanism that allow for the efficient implementation and subsequent assessment of new initiatives. 4. These recommended improvements should enable the NIH SBIR manag- ers to address the four mandated congressional objectives in a more ef- ficient and effective manner. B. The NIH SBIR program is focused on commercialization and has seen significant achievement. Nonetheless, there are also clear opportunities for further improvement. Continued management attention and addi- tional efforts and resources to facilitate commercialization are needed. 1. Commercialization programs. NIH should continue to experiment with commercialization programs, encouraging general application when they show signs of measurable success. Current data indicate that of the 114 companies participating in the Technology Assistance Program in 2004- 2005, 23 had received a total of $22 million in additional funding. Other milestone indicators were also positive. 2. Funding for commercialization programs. Congress should consider updating the current limits on spending for this purpose. The current limit of $4,000 per year per awardee imposes considerable constraints on innovative programming in this area. Consideration should be given to substantially increasing this amount, and the flexibility of its use. C. NIH should adopt a more data-driven culture for its SBIR program, with regular assessment driving policy and program management. The current evaluation efforts at NIH are a good start. Given sufficient additional funding, the Committee recommends: 1. Annual SBIR Program Report. The NIH SBIR Program Coordinator should be tasked with preparing a much expanded annual SBIR Program Report for submission to a new Advisory Board (see E, below). The report should summarize all relevant data about awards, outcomes, and program initiatives and activities. 2. Assessment plan. The program should review its data collection program, identify improvements and develop a formal plan for evaluation and as- sessment. The internal assessment program should be supported by sys- tematic, objective outside review and evaluation of the NIH program. D. NIH should focus greater attention on participation by minority- and woman-owned firms in the program. 1. Encourage participation. NIH should encourage woman- and minority- owned businesses to submit SBIR proposals and track their successes in winning Phase I and Phase II awards.

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH 2. Improve data collection and analysis. Data collection efforts, as noted above, need to be substantially improved, particularly with regard to women and minorities. i. The absence of effective, timely monitoring of minority and woman participation is troubling. This should be corrected on an urgent basis. ii. Further analysis of the data, backed by case interviews, should be undertaken to determine the sources of recent trends and the steps that might be taken to address them. 3. Extend outreach to younger woman and minority students. NIH should encourage and solicit women and underrepresented minorities working at small firms to apply as Principal Investigators and Co-Investigators for SBIR awards and track their success rates. i. Encourage emerging talent. The number of women and, to a lesser extent, minorities graduating with advanced scientific and engineering degrees has been increasing significantly over the past decade, espe- cially in the biomedical sciences. This means that many of the woman and minority scientists and engineers with the advanced degrees usu- ally necessary to compete effectively in the SBIR program are rela- tively young and may not yet have arrived at the point in their careers where they own their own companies. They should be encouraged to serve as principal investigators (PIs) and/or senior co-investigators (Co-Is) on SBIR projects. ii. Track success rates. The Committee also strongly encourages NIH to gather and publish the data that would track woman and minority principal investigators (PIs), and to ensure that SBIR is an effective road to opportunity for these PIs as well as for woman- and minority- owned firms. The success rates of woman and minority PIs and Co-Is are a traditional measure of their participation in the non-SBIR re- search grants funded by nonmission research agencies like NIH and NSF, and should be an appropriate measure of woman and minority participation in the SBIR program. After all, experience as a Principal Investigator or Co-Investigator on a successful SBIR program may well give a woman or minority scientist or engineer the personal confidence and standing with agency program officers that encourage them to apply for SBIR awards and found their own firms. E. The NIH should consider creating an independent Advisory Board that draws together senior agency managers, outside experts, and other stake- holders to review current operations and recommend changes to the program.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS 1. An annual Program Report could be presented to the Board on an annual basis. The Board would review the report, including program progress, management practices, and make recommendations to senior NIH officials in charge. 2. The Board might be assembled on the model of the Defense Science Board. It could include senior NIH staff from the ICs and the Director’s Office, on an ex officio basis, and bring together, inter alia, representatives from industry (including award recipients), academics, and other experts in early-stage finance and program management. F. NIH should support and encourage the use of better tools for quality control and evaluation of the SBIR program. 1. Monitor outcomes. As part of the proposed annual Program Report, the Coordinator should monitor SBIR awards and outcomes across the NIH and each institute should develop a similar and compatible capacity. 2. Suggestions from surveys. As part of future surveys, a particular effort should be made to gather suggestions for future program improvement from survey recipients. 3. Benchmarks. Operational program benchmarks for both process and outcomes should be developed and used to assess program effectiveness at every IC as well as for the program as a whole. 4. Public information. NIH should considerably improve the public distri- bution of information about the program, including recent data on awards and on outcomes. 5. Clear responsibilities. As noted above, the IC management, at the senior level should be responsible for the effective management of each IC-based program and, in cooperation with the SBIR Program Coordinator, share responsibility for serving the needs of both the NIH and the applicants and recipients of SBIR awards. G. NIH should consider ways in which the current approach to SBIR award selection might benefit from more program-specific adaptations. Specifi- cally, there appears to be room for improvements in the following areas: 1. Conflict of interest. NIH should explore means of addressing perceived conflicts of interest within the SBIR selection process. While there are inevitable tensions between the need for expertise on selection panels and the interests of those experts, some applicants have expressed concern that the current honor system may not work effectively to deal with those tensions in all cases. 2. Disclosure. While disclosure of conflicts is mandatory, NIH could con-

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH sider mechanisms for ensuring that such disclosure is as effective as pos- sible. NIH might consider spot-checking disclosure statements to improve compliance and to signal that NIH views compliance as important. 3. Voting. NIH might consider adjusting the voting mechanism, to help en- sure that individual panel members do not exert undue influence on award decisions. Currently, all scores from review panelists are counted; exclud- ing outlier scores might be considered. 4. Oversight. The proposed SBIR Advisory Board should be responsible for addressing these and other issues related to award selection, in conjunc- tion with relevant staff at the Center for Scientific Review (CSR—the NIH Center that manages the selection process for the other IC’s). 5. Commercial review. While the NIH SBIR program has registered sub- stantial commercial success, awardees and agency staff have suggested that there is room for considerable improvement, not least in the way in which selection processes assess commercial potential. The difficulties in- volved in balancing the need for effective commercial review with the risk of conflicts of interest have not been adequately addressed by NIH. The agency should consider adopting pilot programs that could improve the quality and fairness of commercial reviews.39 Possible options include: i. Hiring professional commercialization consultants and attaching them to specific study sections. This option could provide significant additional expertise as a resource for the study sections, without fun- damentally changing the review process. It should be evaluated on a test basis and reviewed for enhanced commercialization outcomes. ii. Adding staff with industry experience. Adding new staff members with significant industry experience in the development and commer- cialization of new products could bring a new dimension to the review and assessment experience. iii. Separating commercial and scientific review processes, with com- mercial review considered by a separate, possibly semi-permanent, panel of commercial experts appointed (or hired) specifically for this purpose. iv. Follow-up assessment. Best practices might be better identified in the selection process by closer analysis of the connection between award outcomes and selection processes. H. NIH is to be commended for its flexible, industry-driven approach to the 39 Improving the commercial review process, which this recommendation addresses, is not the same as enhancing commercial potential as a criterion for successful review.

OCR for page 19
7 FINDINGS AND RECOMMENDATIONS SBIR award process. To improve the program’s operation further, NIH should consider mechanisms for substantially shortening the average time between initial application and cash-in-hand for award winners. 40 1. Strengths of the NIH SBIR award process include: i. Multiple opportunities. In particular, NIH should be commended for providing three application deadlines, rather than the annual deadline used at some other agencies, encourages timeliness, reduces delay, and therefore facilitates participation by microfirms. ii. Resubmission. The availability of resubmission is another important and positive aspect of the NIH program, allowing companies to fix problems with their applications rather than simply rejecting them, as is the practice at other SBIR programs. iii. Investigator-driven applications. NIH’s investigator-driven approach to topics also makes it unnecessary for applicants to wait for the “right” topic to be part of a solicitation. This program flexibility is a major advantage of the NIH program. 2. Notwithstanding these strengths, the NIH SBIR program still faces chal- lenges: Even with these advantages, delays still occur. For example, companies sometimes cannot afford to accept the delays involved in resubmission, and, in some cases, they cannot afford the overall time lags inherent in the full cycle from initial application to cash-in-hand. These delays and uncertainties tend to reduce the effectiveness of the program and should be reduced where possible. 3. Suggested mechanisms for improving the decision cycle include: i. NIH should develop a selection process that is tuned as much as possible to the specific needs of small business. The current award process is tightly intertwined with the selection process for other NIH programs, notably R01. This approach may be entirely appropriate for awards to academic institutions and university faculty, but it is often less appropriate for an award program for small business, where de- lays can in many cases lead firms to abandon promising research. ii. The recent NIH shift to electronic submission is an encouraging development, one that was identified early on in this study. It should help to reduce cycle delays, especially if NIH uses the new system as an opportunity to improve the process as a whole. The 40 Eighty percent of NIH respondents to the NRC Phase II Survey indicated that they had experi- enced a gap between Phase I and Phase II.

OCR for page 19
 SBIR AT THE NATIONAL INSTITUTES OF HEALTH NASA model and DoE’s recent conversion are potential guides to best practice. iii. Quick rebuttal. Numerous winners and applicants stated in inter- views that review panels simply did not understand their applications, or rejected them on questionable grounds. NIH should seek ways to use new technology as the basis for new procedures that would allow a more iterative approach within a single review cycle. Resubmission is not in itself an adequate response to this problem, in light of the substantial delays it imposes on applicants. 41 One approach would be to have NIH change its selection process to make a short written summary from the lead reviewer available electronically to the applicant before the study section meeting. The applicant could then provide a one-page commentary or rebuttal, to be distributed immediately before the meeting. This process might have multiple positive benefits, including improving perceptions of fairness and adding quality control to the selection process. 4. The Committee strongly encourages NIH to experiment with different ap- proaches to selection using the pilot program approach described below. I. NIH should develop a formal mechanism for designing, implementing, and evaluating pilot programs. 1. Need for experimentation. Addressing these concerns will require re- sources and time for experimentation. i. Preserving flexibility. Making changes initially through pilot pro- grams allows NIH to alter selected areas on a provisional basis. A single approach may not work for a program that funds such highly diverse projects with very different capital requirements and very dif- ferent product development cycles. ii. Lowering cost. Pilot programs allow Institutes to investigate program improvements at lower risk and lower cost than through changes to the program as a whole. However, effective pilot programs require rigorous design and evaluation. 2. Program changes need follow-up assessment. Some of the most sig- nificant changes to the SBIR program at NIH—notably changes in award size—have apparently occurred without any evaluation or a clearly articu- 41 Because rejections are received too late for applicants to resubmit during the next submission cycle, an additional delay of 4 months is widely experienced in addition to the actual time needed to review the proposal again.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS lated rationale. Other changes, such as the recent NCI-led commercializa- tion assistance pilot, lack a formal evaluation and assessment component. Performance benchmarks, metrics, and timely evaluation, internal and external, should be included in program modifications. 3. Improving perceptions of fairness. Additional improvements to the pro- gram to address perceptions of unfairness should be considered. These could include more commercial expertise, the right of rebuttal, enhanced use of resubmission, and measures to address perceptions of conflict of interest. 4. Suggested pilot programs. NIH should consider pilot programs designed to shorten the program’s award cycle time to be more commercially rel- evant, refine certain selection processes, and better assess the impact of the trend toward increased award sizes: i. Larger awards. NIH is unique in the extent to which funding has been made available beyond the standard limits set by SBA. This flexibility is both appropriate and valuable. The use of large awards at NIH raises some important questions. NIH staff has offered several different justifications for larger awards.42 None of these rationales has been based on research and assessment of the program, notwithstanding the possible impact of larger awards on the program. At a minimum, NIH should develop a clear justification for these larger awards, based primarily on data drawn from the program or elsewhere, which addresses the range of program risks identified in the program management chapter of this report. NIH should also develop a formal program to review the impact of the larger awards that are already being made. This should include developing a clear rationale, identifying selection criteria for larger awards, and a robust assessment component, including third-party review to monitor outcomes. Because the additional resources used to fund these awards are substantial, the awards need to achieve a specific objective and/or yield significantly different or better outcomes than multiple standard-sized awards using equivalent funding. ii. Direct to Phase II. Some agency staff and recipient companies have suggested that research that is otherwise promising has been excluded 42These include the need to focus resources on the best applications, the high cost of drug devel- opment, the high cost of biomedical research, and the lack of inflation adjustments to the standard award size over the last 10 years.

OCR for page 19
0 SBIR AT THE NATIONAL INSTITUTES OF HEALTH from receiving adequate Phase II level funding because all award recipients have to garner a Phase I first.43 As well, some program participants have suggested that consideration be given to changing the requirement that SBIR recipients apply for and receive a Phase I award before applying for Phase II. They suggest the rigid application of this requirement has the potential to exclude promising research that could help agencies meet their congressionally mandated goals. However, permitting companies to apply directly to Phase II has the potential to change the program significantly. In particular it could shift the balance of both awards and funding significantly away from Phase I toward Phase II. Every additional Phase II award represents approximately 7.5 Phase I awards. If “direct to Phase II” were as attractive to applicants as proponents suggest, it might became a significant component of the program. This in turn could make a very substantial difference to funding patterns in SBIR to the detriment of Phase I.44 Moreover, expanded Phase I awards, such as those now used at NIH, can meet the same need without affecting the structure of the program. Accordingly, this fundamental change to the program structure should not be made. iii. Drug discovery. Given the large size of the sums required, it would be appropriate for NIH to consider a number of possible approaches to the needs of small companies in this area. Some of these approaches may be appropriately housed within the SBIR program. For example, NIH has already experimented with the Competing Continuation Awards program designed to provide funding during the regulatory re- view process. However, NIH should also ensure that efforts to address drug development issues do not negatively affect the SBIR program outside drug discovery. Further review of the program’s role in drug discovery, and its limitations, should be undertaken. J. Additional management resources. To carry out the measures recom- mended above to improve program utilization, management, and evalu- ation, the program will require additional funds for management and evaluation. 43 Discussions with NIH SBIR program managers, June 13, 2006. 44 PhaseI awards may have particular importance in meeting noncommercial objectives of the program, for example, helping academics to transition technologies out of the lab into startup companies.

OCR for page 19
 FINDINGS AND RECOMMENDATIONS 1. Effective oversight relies on appropriate funding.45 An evidence-based program requires high quality data and systematic assessment. Sufficient resources are not currently available for these functions. 2. Increased funding is needed to provide effective oversight, including site visits, program review, systematic third-party assessments, and other nec- essary management activities. 3. To achieve the goal of providing modest amounts of additional funding for management and evaluation, there are three options that might be considered: i. Additional funds might be allocated internally, within the existing budgets of NIH, as the Navy has done at DoD. ii. Funds might be drawn from the existing set-aside for the program to carry out these activities. iii. The set-aside for the program, currently at 2.5 percent of external research budgets, might be marginally increased, with the goal of pro- viding management resources necessary to maximize the program’s return to the nation.46 The key point is that additional resources for program management and evaluation are necessary to optimize the nation’s return on the substantial annual investment in the SBIR program. 45According to recent OECD analysis, the International Benchmark for program evaluation of large SME and Entrepreneurship Programs is between 3 percent (for small programs) and 1 percent for large-scale programs. See Organisation for Economic Co-operation and Development, “Evaluation of SME Policies and Programs: Draft OECD Handbook,” OECD Handbook CFE/SME(2006)17, Paris: Organisation for Economic Co-operation and Development, 2006. 46 Each of these options has its advantages and disadvantages. For the most part, the Departments, Institutes, and Agencies responsible for the SBIR program have not proved willing or able to make additional management funds available. Without direction from Congress, they are unlikely to do so. With regard to drawing funds from the program for evaluation and management, current legislation does not permit this and would have to be modified, therefore the Congress has clearly intended program funds to be for awards only. The third option, involving a modest increase to the program, would also require legislative action and would perhaps be more easily achievable in the event of an overall increase in the program. In any case, the Committee envisages an increase of the “set-aside” of perhaps 0.03 percent to 0.05 percent on the order of $35 million to $40 million per year or, roughly, double what the Navy currently makes available to manage and augment its program. In the latter case (0.05 percent), this would bring the program “set-aside” to 2.55 percent, providing modest resources to assess and manage a program that is approaching an annual spend of some $2 billion. Whatever modality adopted by the Congress, without additional resources the Committee’s call for improved management, data collection, experimentation, and evaluation may prove moot.