An Assessment of the SBIR Program at the Department of Energy (2008)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

Summary I. Introduction The Small Business Innovation Research (SBIR) program was created in 1982 through the Small Business Innovation Development Act. As the SBIR pro- gram approached its twentieth year of operation, the U.S. Congress requested the National Research Council (NRC) of the National Academies to “conduct a com- prehensive study of how the SBIR program has stimulated technological innova- tion and used small businesses to meet Federal research and development needs” and to make recommendations with respect to the SBIR program. Mandated as a part of SBIR’s reauthorization in late 2000, the NRC study has assessed the SBIR program as administered at the five federal agencies that together make up some 96 percent of SBIR program expenditures. The agencies, in order of program size are the Department of Defense (DoD), the National Institutes of Health (NIH), the National Aeronautics and Space Administration (NASA), the Department of Energy (DoE), and the National Science Foundation (NSF). Based on that legislation, and after extensive consultations with both Congress and agency officials, the NRC focused its study on two overarching questions. Three primary documents condition and define the objectives for this study: These are the Legisla- tion—H.R. 5667, the NAS-Agencies Memorandum of Understanding, and the NAS contracts accepted by the five agencies. These are reflected the Statement of Task addressed to the Committee by the Academies leadership. Based on these three documents, the NRC Committee developed a compre- hensive and agreed set of practical objectives to be reviewed. These are outlined in the Committee’s formal Methodology Report, particularly Chapter 3, “Clarifying Study Objectives.” National Research Council, An Assessment of the Small Business Innovation Research Program: Project Methodology, Washington, D.C.: The National Academies Press, 2004. Accessed at <http://books.nap.edu/catalog. php?record_id=11097#toc>.  

 Sbir AT THE DEPARTMENT OF ENERGY First, how well do the agency SBIR programs meet four societal objectives of inter- est to Congress: (1) to stimulate technological innovation; (2) to increase private sector commercialization of innovations; (3) to use small business to meet federal research and development needs; and (4) to foster and encourage participation by minority and disadvantaged persons in technological innovation. Second, can the management of agency SBIR programs be made more effective? Are there best practices in agency SBIR programs that may be extended to other agencies’ SBIR programs? To satisfy the congressional request for an external assessment of the pro- gram, the NRC conducted empirical analyses of the operations of SBIR based on commissioned surveys and case studies. Agency-compiled program data, pro- gram documents, and the existing literature were reviewed. In addition, extensive interviews and discussions were conducted with program managers, program participants, agency ‘users’ of the program, as well as program stakeholders. The study as a whole sought to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program’s contribution to accomplishing agency missions. (See Box S-1, which highlights findings from the case studies on how SBIR companies commercialize.) To the extent possible, the evaluation included estimates of the benefits (both economic and noneconomic) achieved by the SBIR program, as well as broader policy issues associated with public-private collaborations for technology development and government support for high technology innovation. Taken together, this study is the most comprehensive assessment of SBIR to date. Its empirical, multifaceted approach to evaluation sheds new light on the operation of the SBIR program in the challenging area of early stage finance. As with any assessment, particularly one across five quite different agencies and departments, there are methodological challenges. These are identified and dis- cussed at several points in the text. This important caveat notwithstanding, the scope and diversity of the report’s research should contribute significantly to the understanding of the SBIR program’s multiple objectives, measurement issues, operational challenges, and achievements. This volume presents the Committee’s assessment of the SBIR program at the Department of Energy. These congressional objectives are found in the Small Business Innovation Development Act (PL 97-219). In reauthorizing the program in 1992 (PL 102-564), Congress expanded the purposes to “emphasize the program’s goal of increasing private sector commercialization developed through fed- eral research and development and to improve the federal government’s dissemination of information concerning small business innovation, particularly with regard to woman-owned business concerns and by socially and economically disadvantaged small business concerns.” See, for example, Box 1-1 in Chapter 1 of this report, which lists the multiple sources of bias found in large innovation surveys. 

SUMMARY  Box S-1 How SBIR Companies Commercialize— Findings from the Case Studies Companies interviewed for this study illustrate some of the many approaches to commercialization taken by DoE SBIR recipient firms. The case studies of these companies can be found in Appendix D. One company marketed the technology developed with its lone project, sub­ contracted production, and achieved huge revenues (Atlantia). Another marketed and manufactured its own product (IPIX), eventually going public. Three companies are manufacturing products based on their SBIR work, while they seek larger partners in order to expand markets (NanoScience, NexTech, Thunderhead Engi­ neer­ing). Other companies achieve commercialization by spinning off (Creare) or licensing (Eltron) their SBIR technologies. Yet another used SBIR research to c ­ reate a new market—in one instance for its core technology (Diversified Tech­ nologies Inc.) and, in another, for an R&D contracting business (PPL). Although some case study companies are currently dependent on SBIR as a major source of their revenues (Airak, Creare, NanoScience), all are actively engaged in commercialization, selling either products or services. Most companies use patents to protect their intellectual property. II. SBIR at the Department of Energy The DoE’s SBIR program is located administratively within its Office of Science (SC). The Office of Science funds research in the basic energy sciences, biological and environmental research, fusion energy sciences, high energy and nuclear physics, and computational science. With an annual budget of approxi- mately $3.5 billion, the Office of Science is the largest federal sponsor of mate- rials and chemical sciences research within the federal government. And with a budget of $104 million in FY2005, the DoE SBIR program is the nation’s third largest. (See Figure S-1.) Given that federal agencies are barred from using any of their SBIR budgets to fund the administrative costs of the program, DoE officials often regard SBIR administrative expenses as an additional, though unspecified, tax.  As host for the SBIR office, DoE’s Office of Science is responsible for the direct costs of administering the program: salaries for the federal employees, support services contracts, and costs such as developing and maintaining the electronic grant management system. Because the Office of Science must use its own funds to administer the SBIR program, and because of the historical resistance to SBIR at DoE, there has been a tendency for the Office of Science to limit the resources for administering the Public Law 102-564. 

 Sbir AT THE DEPARTMENT OF ENERGY Other (3.3%) * NSF (4.3%) $79 million Total = $1.85 billion * NASA (5.6%) $103 million * DoE (5.6%) $104 million * DoD (50.9%) $943 million * HHS (30.4%) $562 million FIGURE S-1  Dimensions of the SBIR program in 2005. SOURCE: U.S. Small Business Administration. Accessed at <http://tech-net.sba.gov/>. program—ultimately to the point where 1.5 full-time DoE employees, assisted by five contractors, were managing the administration (though not the review) of approximately 1,500 applications annually (both Phase I and Phase II). In March 2005, DoE SBIR reported a staff of three federal employees and six con- tractors. They still reported substantial resource constraints, including the use of 20-25 percent of their time on redundant paperwork issues. In order to cope with these resource constraints, the SBIR staff has placed a high premium on sticking to a clearly mapped-out schedule and meeting estab- lished deadlines. Low funding figure S-1 levels for administration mean however that the DoE SBIR staff devotes nearly all their time to managing the processes for gen- erating technical topics, and for receiving, evaluating, and selecting grant applica- tions. This leaves little time for activities such as outreach, measuring Phase III activity, encouraging Phase III activity (both within and outside the Department, including the national laboratories), internal evaluation, strategic planning, and the documentation of successes. Despite these resource constraints, the NRC committee found that the Depart- ment of Energy has made significant progress in meeting the legislative objectives of SBIR and that the program is addressing the mission of the Department of Energy. As noted below, the committee has also identified a set of related recom- mendations that are designed to improve the operation of the SBIR program at the Department of Energy. III. Summary of Key Program Findings The DoE SBIR program is making significant progress in achieving the congressional goals for the program. The SBIR program is sound in concept and 

SUMMARY  Box S-2 The Mission of the Department of Energy DoE’s mission is “to advance the national, economic, and energy security of the United States; to promote scientific and technological innovation in support of that mission; and to ensure the environmental cleanup of the national nuclear weapons complex.” The agency has identified four strategic goals that support achieving this mission: Defense Strategic Goal. Protecting national security by applying advanced science and nuclear technology to the nation’s defense. Energy Strategic Goal. Protecting national and economic security by pro­ moting a diverse supply and delivery of reliable, affordable, and environmen­ tally sound energy. Science Strategic Goal. Protecting national and economic security by providing world-class scientific research capacity and advancing scientific knowledge. Environment Strategic Goal. Protecting the environment by providing a r ­ esponsible resolution to the environmental legacy of the Cold War and by pro­ viding for the permanent disposal of the nation’s high-level radioactive waste. SOURCE: Department of Energy. effective in practice at DoE. With the programmatic changes recommended by the Committee in this report, the SBIR program should be even more effective in achieving its legislative goals. Overall, the program has made significant progress in achieving its congressional objectives by: • Stimulating technological innovation. o Generating patents and publications. The SBIR program at DoE supports knowledge transfer in several ways. A significant number of the projects responding to the NRC Phase II Survey (43 percent) reported at least one patent application. Of these, over a third (37 per- cent) reported receiving a patent related to the SBIR project. In addi- tion, nearly half (or 46 percent) of projects surveyed by NRC resulted in at least one peer-reviewed article. See Finding G on “Knowledge Creation and Dissemination Effects” in Chapter 2. Without detailed identifying data on these patents and publications, it is not feasible to apply bibliometric and patent analysis techniques to assess the relative importance of these patents and publications. 

 Sbir AT THE DEPARTMENT OF ENERGY o Stimulating the transfer of technology from universities to the market. The NRC Phase II survey also suggests that SBIR awards are supporting the transfer of knowledge, firm creation, and partnerships between universities and the private sector: At DoE, just over one- third of projects had some alignment with a university, through the use of university faculty as contractors on the project, use of universities as subcontractors, or employment of graduate students o Indirect paths. Case studies conducted for this study provide ­anecdotal evidence concerning beneficial “indirect path” effects—that ­ projects provide investigators and research staff with knowledge that may later become relevant in a different context—often in another project or even another company. • Using small businesses to meet federal research and development needs. o Mission alignment. The DoE SBIR program is operated in alignment with the department’s mission. SBIR awards at DoE appear to be selected primarily on the basis of their potential to advance knowl- edge and provide solutions in the field of energy. There is also no evidence that DoE awards are made in fields not directly linked to the department’s mission. o Potential for collaboration with the National Laboratories has not been fully realized. The National Laboratories, while an important source of technical reviewers for SBIR proposals, are not strongly involved in the SBIR program. As a result, the potentially signifi- cant role of the National Laboratories as partners with SBIR-award- r ­ ecipient firms does not appear to be fully realized. o Encouraging small business formation. The SBIR program at DoE has provided significant support for small business, frequently acting as the impetus for the foundation of new firms. Nearly one-fourth of survey respondents with DoE awards indicated that their companies were founded entirely or partly because of an SBIR award. o Providing a catalyst for research. SBIR awards encourage research projects that would not otherwise have been undertaken. More than 80 percent of DoE NRC Phase II Survey respondents reported that they See Table 4-14 in National Research Council, An Assessment of the Small Business Innovation Research­ Program, Charles W. Wessner, ed., Washington, D.C.: The National Academies Press, 2008. See related Findings C and D in Chapter 2. Among case study firms, networking was identified as critical when attempting to find partners to assist with commercialization. Possibly the most dramatic identified success from the program has been Atlantia’s SeaStar® technology, a cost-effective alternative for the development of otherwise inaccessible small oil fields in deep water. The technology generated more than $500 million in revenues for Atlantia. 

SUMMARY  would likely or certainly not have gone forward with their project in the absence of SBIR funding.10 o Partnering and networking. SBIR further facilitates technological innovation through the creation of new partnerships and the strengthen- ing of networks of innovators.11 Innovative small businesses also use SBIR to fund alternative development strategies, exploring technologi- cal options in parallel with other activities. o Certification effect. Receipt of SBIR awards can, in addition, provide a “stamp of approval,” allowing smaller companies to better access large private sector companies, including government contractors, and attract private investors.12 • Fostering and encouraging participation by minorities and disadvan- taged persons in technological innovation.13 o Consistent support, but no long-term trend is apparent. Data from SBA indicates that between 1992 and 2005, the Phase I share of woman- and minority-owned businesses at DoE averaged just over 20 percent, with a decline in the early years of this decade, followed by an increase, accelerating in 2005. (See Figure 2-1.) Data for Phase II are similar, although slightly lower. On average, woman- and ­minority- owned firms won 22.1 percent of Phase I awards from 2001-2005, and 19.1 percent of Phase II awards.14 o Lagging application success rates. For Phase I, applications from woman-owned businesses have had a lower rate of success compared to all other groups—by approximately 3 to 10 percentage points—in every year except one. For minority-owned companies, the success rate is better than for woman-owned companies, but still lags behind the “other” category (neither woman-owned nor minority-owned). For 2002-2003, the success rate of minority-owned businesses was consid- erably lower than that for woman-owned and all other businesses. 15 • Increasing private-sector commercialization derived from federal research and development.16 10See Figure 7-1. 11See Figure 7-2. 12For an early reference to the certification effect, see Joshua Lerner, “Public Venture Capital: Rationales and Evaluation,” in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, Charles W. Wessner, ed., Washington, D.C.: National Academy Press, 1999. 13See related Finding E in Chapter 2. 14See Chapter 6. 15See Chapter 6. 16See related Finding B in Chapter 2. 

 Sbir AT THE DEPARTMENT OF ENERGY o Significant commercialization. A significant percentage of DoE SBIR projects are commercialized to some degree. The NRC Phase II sur- vey data indicate that 41 percent of SBIR-funded projects reach the marketplace or have commercialization underway.17 Of the DoE SBIR award recipient firms that responded to the NRC Phase II Survey and reported sales of some type, 76 percent sold to domestic private sector firms and 14 percent to export markets.18 The NRC Phase II Survey data also show that a much smaller num- ber (4 percent) of projects generate more than $5 million in revenues.19 However, as in cases where the market is inherently limited—as with sensitive energy technologies—products developed with DoE SBIR assistance often cannot become large commercial successes.20 o Commercialization support. DoE has a strong and innovative history of commercialization support, and has expanded its efforts in recent years. DoE has sponsored a Commercialization Assistance Program (CAP) for the past 17 years. That program has provided, on a voluntary basis to Phase II awardees, individual assistance in developing busi- ness plans and in preparation of presentations to potential investors. For the past 6 years, DoE has also offered its Phase II awardees addi- tional ­market identification and networking services.21 These programs provide professional assistance in business plan development and/or market evaluation to participating SBIR companies. IV. Summary of Key Program Recommendations The NRC committee’s recommendations are designed to improve the already effective operation of the SBIR program at the Department of Energy. They complement the core findings that the program is addressing its legislative goals. With the programmatic changes recommended here, the SBIR program should be even more effective in achieving its legislative goals. • Improve program processes. o Provide pre-release information on topics. Pre-proposal commu- nication through a fair, transparent mechanism is likely to improve proposal quality and, therefore, overall program effectiveness. The 17See Figure 4-1. Twenty-four percent of NRC Phase II Survey respondents reported products/ services/or processes in use at the time of the survey, and 18 percent reported commercialization underway (figures rounded). See NRC Phase II Survey, question 1. 18See Table 4-2. 19See NRC Phase II Survey, Question 4b. 20The case study of Diversified Technologies illustrates this phenomenon with regard to specialized transformers. See Appendix D. 21DoE SBIR publications and Web site; interviews with DoE SBIR staff. 

SUMMARY  Department of Energy might consider the DoD Pre-Release informa- tion exchange model, whereby the relevant technical officer for each topic is available via email and phone for questions during a period before the official release of the solicitation.22 o Develop a match-making function for SBIR awardees.23 The depart- ment’s SBIR program should bring SBIR participants together with potential corporate customers, perhaps in trade show, technical chal- lenge workshop, or technology demonstration/validation formats. These functions could include large corporations identified by the agency’s two SBIR commercialization assistance vendors, including large energy technology corporations that serve as DoE contractors. o Rationalize funding allocations.24 The department might consider whether current ad hoc funding allocations—based on the origins of the SBIR funding by program rather than according to project quality and need—are sufficient, or whether a policy that allows individual DoE programs to benefit from SBIR projects in excess of their con- tributed amounts would prove more effective. • Encourage collaboration with National Laboratories.25 o DoE should encourage the National Laboratories to participate as subcontractors to the small business on SBIR projects, not least by removing regulatory barriers to the use of National Laboratories as subcontractors. DoE should also develop procedures to track the rela- tionship between National Laboratories and the SBIR program more formally, including the documentation of Phase III successes. • Increase participation by woman- and minority-owned firms.26 o DoE should undertake an assessment of the participation rates of woman- and minority-owned firms in its SBIR program, and identify strategies to improve their success rates. The development of outreach efforts and other strategies should be based on empirical analysis of past proposals and feedback from the affected groups.27 22See related Recommendation A in Chapter 2. Since 2006, the relevant DoE technical officer has been available via email up to the closing date of the announcement. 23See related Recommendation B in Chapter 2. 24See related Recommendation C in Chapter 2. 25See related Recommendation D in Chapter 2. 26See related Recommendation E in Chapter 2. 27This recommendation should not be interpreted as lowering the bar for the acceptance of ­proposals from woman- and minority-owned companies, but rather as assisting them to become able to meet published criteria for grants at rates similar to other companies on the basis of merit, and to ensure that there are no negative evaluation factors in the review process that are biased against these groups. 

10 Sbir AT THE DEPARTMENT OF ENERGY • Produce enhanced reports on the program for DoE management and the Congress. o DoE should provide Congress with a summary annual report on the SBIR program. This should include descriptive statistics for applica- tions, awards, and outcomes along the dimensions identified in this report, including knowledge creation, technology innovation, and impact on agency mission, as well as commercialization. • Conduct regular internal and external assessments.28 o As part of this assessment process, DoE should produce regular reports from the commercialization database. In addition to improved internal assessment capabilities that can be used to enhance program opera- tions, DoE should also commission regular external arms-length evalu- ations to assess the program progress. • Consider the creation of an advisory board.29 o DoE should consider the creation of an independent advisory board that draws together senior agency management, SBIR managers, and other stakeholders as well as outside experts to review current opera- tions and achievements and, as appropriate, recommend changes to the SBIR program. • Provide additional management funding.30 o Managing what should be a data-driven program requires high ­quality data and systematic assessment, which in turn requires appropriate funding. Increased funding is also needed to provide effective over- sight, including site visits, program review, systematic third-party assessments, and other necessary management activities. 28See related Recommendation F in Chapter 2. 29See related Recommendation H in Chapter 2. 30See related Recommendation G in Chapter 2.