National Academies Press: OpenBook

SBIR Program Diversity and Assessment Challenges: Report of a Symposium (2004)

Chapter: Panel VI — Roundtable Discussion

« Previous: Panel V — SBIR at the National Science Foundation
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

Panel VI
Roundtable Discussion

Moderator:

Jacques S. Gansler

University of Maryland

Dr. Gansler said that a great range of viewpoints had been expressed by symposium participants, which was to be expected. “This is the kickoff session, not the final session, and our challenge is to address all of these issues.”

He highlighted several of those points, beginning with the need to link federal and state initiatives. He also expressed the desirability that some of the smaller SBIR agencies participate in the study, at least in terms of input and recommendations. “The very worst thing,” he said, “would be to end up with recommendations that have an adverse effect on the smaller agencies.” He said that the focus would necessarily be on the five largest agencies, which receive 96 percent of the SBIR dollars, but noted that the small ones may have some good practices to learn about “because they may be more nimble and faster to move.”

He turned to the Department of Energy’s discussion of proprietary data. Accessing this data, he said was “an issue we clearly need to address. We need to get that data from the agencies. I see no reason why we cannot protect it—either by not identifying the company or by dealing with aggregate data. In past studies that has not been a problem.”

Pay Attention to Agency Differences

He said that a number of people had asked why this study would focus on individual agencies. “It strikes me,” he answered, “that each of the agency missions is so dramatically different that we have to be aiding the mission of each agency. If we don’t pay attention to that, we are missing the number one priority, which is to be sure these resources are being spent with regard to mission needs.”

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

To this end, he said, subcommittees of the steering group were planned for each agency. At the same time, it would be important for all members to have an overview of all the agencies. “The study panel and steering committee also need ‘cross-pollination’,” he said, “and in the overall activity we need to be able to see the big picture and its effectiveness and value to the nation.” He said the panel would look for best practices to draw on, pointing at the effectiveness of DoE in doing “so much in three months.”

He concluded that “this is going to require a great deal of input from a lot of people,” and that it was a “challenging kind of mission, and always has been—how you demonstrate the value of an R&D project.” He said that it was never possible to do so with precision, but it was possible at least to design better approaches.

Jon Baron

Coalition for Evidence-Based Policy

Mr. Baron noted that the symposium had heard examples of “the program at its best, of technology from companies whose SBIR-developed products have contributed to our lives in important ways.” He referred to the cases of Martek Biosciences and Advanced Technology Materials. Technology developed by the latter company, he said, made it possible to transport toxic gases safely and avoid the need to evacuate whole communities should a transport truck crash.

Needed: Help with Commercialization

He said that a company needed to have two types of skills to develop such life-improving innovations. The first is technical expertise—that is, the ability to develop a sound technical idea and to carry out high-quality research. The other skill is entrepreneurship, the ability to translate a good idea into a prototype, to describe its commercial value in a compelling way to potential investors or customers, and ultimately to convert it into a commercially-successful product sold to commercial or military customers. These two steps required different skills, he said. Government, through the SBIR program, was good at evaluating the first skill—technical capability. But the government was not as good at identifying companies with the second skill—commercialization, and many agencies did not even attempt to find and fund true entrepreneurs who have that skill.

Addressing the importance of entrepreneurial talent, he said that a typical venture capital firm preferred to bet on first-rate management than on first-rate technology. This approach is substantially different from that of a government agency, he said. Instead of expecting this function from government, he suggested, it might be more effective to develop proxies or certain techniques that add this function. As examples he mentioned further investigation of the NSF

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

“Phase II-B” and the DoD Fast Track programs, which encourage independent third-party investors to contribute matching cash at the stage of commercialization. He suggested that the panel might investigate whether the presence of such outside support should be used as a criterion for choosing candidates for Phase II or for a larger Phase II award.

Another way to focus funding on entrepreneurs might be to weigh a company’s commercialization track record; it might have won numerous SBIR awards, but what products or sales had come out of that? One important contribution of the study panel, he concluded, could be to suggest ways for government to improve the SBIR program and focus its funding more directly on companies most likely to make large contributions to the agency and the economy.

Paula E. Stephan

Georgia State University

Dr. Stephan said that her comments about the SBIR program stem from her interest in what leads scientists and engineers to be productive and the role that federal funding plays in fostering productivity.

First, she said, her work leads her to expect the benefits of the SBIR program to be highly skewed and distributed unevenly. When scientists and engineers are studied at the individual level, she said, productivity is highly skewed in terms of publications, citations to publications, patents, and citations to patents. This a well-known characteristic of discovery. We also find that productivity distributions are highly skewed at the institutional level, as well. The National Science Foundation’s Science and Engineering Indicators show that a handful of universities hold the lion’s share of patents and royalties and have a lion’s share of R&D expenditures. ISI data indicate that publications are highly skewed across universities as well. In the for-profit sector, we know that many R&D projects are initiated, but few of them actually reach commercialization. The “extreme” case is seen in drug discovery, where the odds of a product reaching the market are exceptionally low. Thus, we would expect the distribution of discovery and innovation to be skewed in the SBIR program as well, and it is important to remember this during program evaluation.

A second point to remember during the study, she said, is that “today’s dead end often can be a key to tomorrow’s success, or what is seen as a failure today can be seen as a success tomorrow.” She referred to the example given by Dr. Moore in his experience with optoelectronics, where a seeming dead end later opened up the way to producing a highly profitable technology.23

Finally, she reinforced the importance of a point made by several speakers—that human resources and training opportunities are critical outcomes of the SBIR program, just as they are of other federal investments in discovery.

23  

See opening comments by Duncan Moore in Panel III of the Proceedings.

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

Michael Borrus

The Petkevich Group, LLC

Mr. Borrus said he wanted to list three challenges that would be faced by the committee.

Defining Meaningful Metrics

The first was the challenge of defining meaningful metrics for evaluating the SBIR program, especially metrics that allow comparative evaluation of different agencies and their programs. This, he suggested, would be difficult but not impossible. He said that the panel had heard useful suggestions for metrics that allow a fair evaluation of success. But he added that that the metrics used would depend on how the problem was defined. For example, the success of Martek Biosciences had stimulated the desire to find more entrepreneurs like Mr. Linsert, who were capable of creating and leading a successful company. While good leaders are one source of success, an alternative and equally valid route was to catalyze small businesses that are already successful at developing one product but lack the resources to enter a new market. “We need to be sensitive to what we are excluding by the measures we use,” he said, “as well as to what we are including.”

Learning from other Agencies

A second challenge was to improve program performance as a whole. “There’s a tremendous amount to be learned from what each agency has to teach the others, ” he said, seconding Dr. Gansler’s assertion. He said that NASA, for example, had done strong work with technical and commercial assistance; DoD with self-assessment; DoE with criteria to evaluate Phase III and to speed up the process as a whole; and NIH in using NIAID’s infrastructure to support clinical outcomes. “I hold a somewhat heretical view,” he said, “that the best ways to improve the program are more likely to come from what’s already present and working in the program than from external benchmarks or insights we might bring from outside the program.”

Remaining Relevant

The third challenge, he said, was a contextual challenge: where and how does SBIR fit (1) in the context of federal technology programs overall, and (2) in the larger federal effort to have a significant impact on the long-run economic performance of the United States. He closed by listing a range of questions opened up by that contextual issue:

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
  • In evaluating what the program is, are we missing opportunities for what it could be?

  • Can there be partnerships or trans-agency collaboration in areas like biodefense?

  • Are there opportunities for which SBIR is uniquely suited but under-exploited?

  • In emphasizing SBIR, could we be unintentionally limiting small businesses’ ability to make other positive impacts on federal agencies and their missions?

Linda F. Powers

Toucan Capital Corporation

Ms. Powers said that her perspective was a “pure, unadulterated, investor capitalist viewpoint.” While that perspective was perhaps unusual, she said, it fit with the SBIR program.

Concentrating on the “Valley of Death”

Her fund was one of the largest and only funds that invested in technology companies at the stage where research was coming out of the lab and into the beginnings of commercialization. That meant, she said, that her company concentrated on what was referred to as the “valley of death”—the stage between the development and marketing of a product. “It’s a very fragile stage,” she said, “where companies are going to be set on a path of success or non-success.”

From the viewpoint of her firm, she said that the choice of metrics was crucial. She preferred to evaluate a program not by how many papers or patents emerged, but by true commercialization, which is indicated by sales. The reason for emphasizing sales, she said, was that it was a reliable measure of the utility of an innovation in meeting an unmet need. Sales were also simple and easy to measure.

Sales as a Reliable Metric

Her next comment was about company reporting. She said it was appropriate to be sensitive about redundancy and work burdens, but that reporting was essential—knowing the outcome of a project and being able to measure it. Here again she advocated a simple approach: was the company still alive and making any sales, and was any portion of its product mix related to a product or process specified in the SBIR grant? This approach was not perfect or scientific, she said, but “sort of meets the 80 percent rule of significance” and indicated whether the SBIR had something to do with a company’s success today.

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

She noted the comment that one size does not fit all, and that this study should appreciate and focus on the differences among agencies. The time frame for an NIH product was different from that of a DoD product, she said, and the posture of a small business in a defense contract was different from that of a small biotech firm vis-à-vis “big pharma.” The defense contractor was expected to meet specifications, while the biotech firm was expected to innovate.

Evaluating Disruptive Technologies

She turned to “what the program could be,” and commended the earlier discussion about disruptive technologies. “But then we didn’t talk much about that afterward,” she said, adding that if the reviewers really wanted disruptive technologies, “we have to re-look at some operational aspects for at least a small slice of the program.” She suggested that peer reviewers who were established, prominent people in a technical field might not be the best people to evaluate disruptive technologies or paradigm shifts.

Next she suggested that more interaction between the investor community and the SBIR program was “long overdue.” The investor community, she said, attaches validation to programs that are selected for grants, but only a few parts of agency programs, such as DoD’s Fast Track, attach validation to what private investors are funding. “I’d like to see more of that,” she said. “I’d like to see business people on the review committees, more venture people on evaluation panels, and generally more direct contact and involvement with the investor community.”

Small Companies Need Consistent Funding

As her last observation, she encouraged the panel to look at ways to make the program more effective in regard to commercialization per se. Time cycles, she said, are vital. The example of the DoE evaluating its program in three months is excellent, she said, but the value of an annual evaluation to investors is limited. “It is much more important for grants to be done through the year on a rolling basis, rather than quickly on a far-apart basis,” she said. “The reason so many companies die trying to get across the valley of death is that it is so wide and takes companies so long to cross it.” A lot of that time is spent waiting for grant cycles and other agency procedures, she said. In the biotech world, each day that passes costs a company $3 to 5 million to develop a major product; for a small biotech firm, the product cost is about $1 million a day. Venture capital companies have to consider costs constantly. A Phase II SBIR award of $750 thousand is expected to last for 2 years, but such grants, she said, are too small to support a company during delays. “Even in an early stage,” she said, “companies are burning that much per month.”

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

Finally, she said, intellectual property is “absolutely key”—a fact that is not widely appreciated. She praised the recommendation by the Department of Commerce that universities should start receiving IP rights. That “would be best from an investor’s perspective,” she concluded, while representing the kinds of practicalities that investors need to see.

James Turner

House Science Committee

Mr. Turner said he had only a few points to add to what Ms. Powers had said. First, he urged the study panel to take full advantage of its opportunity to do a thorough evaluation of the SBIR program. It is a rare luxury, he said, to spend two-and-a-half or three years on a report, unrestrained “by quarterly reports, annual appropriations, or a two-year election cycle. You have time actually to think about something, and I hope you’ll take the time to do it right.” He urged the panel to come up with a metric that could be used both by agency managers as the basis for operational improvements and by the Congress for legislative improvements.

Looking at all the Agencies

Second, he reaffirmed the importance of looking at all of the agencies, including those with relatively small SBIR programs. He said he had learned from a representative of the Department of Agriculture, for example, that its distribution of grants should be different from those of the DoD or NIH; for agriculture, it is very important that the SBIR program has a strong rural component. He also reminded the panel members that measures to solve problems in the largest agencies might be effective there, but turn out to distort or harm the programs of smaller agencies. “A cold in the Department of Defense can be pneumonia for the Departments of Commerce and Education,” he said. “So there is a reason to look at all of them in some way.”

He concluded by pointing out that the symposium had drawn excellent attendance from representatives of every sector participating in the SBIR program, including congressional staff, agency representatives, and the small business community. “You’re lucky to have a constituency that really cares,” he told the panel. “I think you can count on them to be supportive of this study and to participate as it goes along.”

• • •

Dr. Gansler closed the symposium by thanking the panel members and participants for taking part, and by expressing optimism about the success of the meetings to come.

Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 138
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 139
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 140
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 141
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 142
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 143
Suggested Citation:"Panel VI — Roundtable Discussion." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 144
Next: Appendix A: Biographies of Speakers »
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In response to a Congressional mandate, the National Research Council conducted a review of the Small Business Innovation Research Program (SBIR) at the five federal agencies with SBIR programs with budgets in excess of $100 million (DOD, NIH, NASA, DOE, and NSF). The project was designed to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. The first in a series to be published in response to the Congressional request, this report summarizes the presentations at a symposium convened at the beginning of the project. The report provides a comprehensive overview of the SBIR program’s operations at the five agencies responsible for 96 percent of the program’s operations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!