National Academies Press: OpenBook

SBIR Program Diversity and Assessment Challenges: Report of a Symposium (2004)

Chapter: Panel IV — SBIR at the Department of Energy

« Previous: Panel III — SBIR at NASA
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

Panel IV
SBIR at the Department of Energy

Moderator:

Patricia R. Forbes

Senate Committee on Small Business and Entrepreneurship

Ms. Forbes said that from the perspective of the Small Business Committee, the SBIR program is “very popular” and has a good deal of political support. Many programs are popular in the states where they do well, she pointed out, but not as popular in other states. This is not true for SBIR, where every state wants “a piece of the program. If they don’t have a piece, they want to figure out how to get one.”

She said that in her opinion, the study panel and its evaluation process were very important. As she introduced the first speaker, she added that her committee hoped to learn much from the panel’s findings that could be applied to the program as best practices.

ACHIEVEMENTS, OPPORTUNITIES, AND CHALLENGES

Milton D. Johnson

Office of Science, Department of Energy

Dr. Johnson said that he would provide an overview of the SBIR program at DoE and his relationship to it. He said that he was Deputy Director for Operations in the Office of Science (SC), “and that means I handle just about everything except the research programs.” The Office had ten laboratories across the country, two operations offices, and a budget of about $3.5 billion.

SC was charged with managing SBIR for the department. In addition to the Office of Science, five other DoE programs received services from the SBIR

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

office: Fossil Energy, Energy Efficiency and Renewable Energy, Nuclear Energy, Environmental Management, and Defense Nuclear Nonproliferation. SC was by far the largest program participating in SBIR, receiving 64 percent of the SBIR budget. Some areas of the department were exempt by law and did not contribute to SBIR, including the naval reactors and weapons programs.

Importance of the DoE SBIR

He said that the SBIR program was important to him for two reasons. First, its annual budget was large—close to $100 million. Second, he served as the chairman of the SBIR Advisory Board, an oversight committee with policymaking authority. This board, which reported to the Director of the Office of Science, allowed all of the DoE program areas to have input into the management and direction of the SBIR program, and ensured that the program was responsive to department needs. He said that the program had proven to be responsive, bringing small businesses into partnership with the national laboratories, universities, and large companies as research and development performers that made important contributions to the department’s mission.

Goal 1: Utilizing Small Businesses in Federal R&D

Despite some early growing pains (i.e., the set-aside itself met with initial resistance), he said that the program had blended well with the rest of DoE’s funding mechanisms (the other 97.5 percent of the budget). SBIR was regarded within the department like any other R&D program, namely, as a vehicle by which the department could accomplish its R&D objectives. The difference was simply that the work was performed by small businesses instead of national laboratories or universities.

This management “blending-in,” he said, was critical for the SBIR office’s ability to manage the program and to gain the cooperation of the technical programs. Without their cooperation, it would be impossible to evaluate the 1,200 grant applications (Phase I + Phase II) received each year. This cooperation was achieved through a balance of both centralized and decentralized management:

  • The SBIR office was centralized in setting common schedules, procedures (for receipt and evaluation of grant applications), and scoring practices. The SBIR office also handled all logistics, such as dealing with thousands of outside peer reviewers.

  • It was decentralized in that the programs were responsible for identifying technical topics, identifying peer reviewers, and selecting grant applications for funding.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Gaining a Full Return on Investment

The best way to gain cooperation from the technical programs was to ensure that each program reaped a full return on its SBIR investment. The best way to achieve this was to approve only those grant applications that received scores above a high threshold.

In this way, the program benefited from a wide range of applicants. He described an example from an area that “might seem too esoteric”—the High-Energy Physics program, which supported complex science projects at accelerator facilities at Berkeley, Stanford, Fermi Lab, Brookhaven, and other locations throughout the world. Even these projects have found a place for small businesses to make a contribution—by developing technology and instrumentation needed to operate such facilities, such as detectors, particle sources, accelerating structures, and power supplies. Small businesses had responded not only by developing instrumentation but also by identifying commercial opportunities for the underlying technologies.

For example, one SBIR company developed a high-power modulator leading to a demonstration of the world’s first high-voltage solid-state switch. In turn, this patented switching technology led to two R&D 100 awards, hundreds of delivered systems, and over $20 million in sales. He concluded that many kinds of SBIR projects were making important contributions to DoE missions, and encouraged the study panel to contact program managers throughout the department to gather views on SBIR quality.

Goal 2: Increasing Commercialization from Federal R&D

Dr. Johnson said that the ability of small businesses to achieve excellent science was not the only measure of success. SBIR was a dual-purpose program, and in addition to increasing the involvement of small businesses in federal R&D, it sought to increase private-sector commercialization of innovations derived from federal R&D (i.e., SBIR Phase III). He said that the DoE had made significant progress with respect to this objective as well.

All DoE Phase II awardees were required to report on their Phase III activity as a condition of their grant. The department learned, through trial and error, an effective way to ask awardees about Phase III. The key, they found, was not to ask about individual projects. A project-by-project approach was tempting because (1) the agencies award and track individual projects and (2) the GAO, in its earlier studies of SBIR, also used this approach. However, it had proved to be imperfect because small businesses do not track their success in this way. Attempts to force them to do so resulted in distorted Phase III data.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Tracking Success as Business Does

In reality, small companies track their success by the products and services derived from SBIR projects. Therefore, DoE learned to ask companies to (1) list all products and services that were derived from their DoE SBIR projects, (2) report on both sales and/or Phase III investment related to these products and services, and (3) identify which Phase II projects contributed to the development of the products and services.

Several years ago, when this type of information was last analyzed, the department learned that approximately 70 percent of its Phase II projects led to Phase III activity, with 50 percent leading to sales of products or services derived from SBIR research. It also learned that the companies achieved approximately $1 billion in Phase III funding (sales or further development funding), approximately three times the support that had been provided by the DoE SBIR program.

He suggested that the biggest contribution the study committee could make would be an independent assessment of Phase III activity in all the agencies. At the same time, he said, it was likely that any attempt to obtain this information by asking companies for Phase III funding data related to individual SBIR projects would produce misleading results.

Help in Developing a Business Plan

In the Commercialization Assistance Program (CAP), Phase II companies were helped to develop a business plan by a contractor with experience in this field. The business planning process involved a “large dose of market research” in which the companies identified and contacted potential customers to learn of their needs. This allowed them to develop a focused business plan responsive to customers’ needs. The contractor also coached the awardees in developing and making presentations to sources of capital. The CAP culminated in a Commercialization Forum in which the small businesses presented their business opportunities to a group of potential investors, including venture capitalists and representatives of large firms.

The program was intense, requiring a time investment by Phase II awardees of approximately 300 person-hours per company. However, it was judged to be very successful, because more than 50 percent of the small businesses participating in the CAP achieved Phase III funding of over $400 million.

The Department of Energy was the first agency to provide commercialization assistance to its SBIR awardees. Recognizing the value of this program, some agencies had “piggybacked” on the CAP while others were studying it as a model, with the assistance of DoE.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Successes of the Program

Dr. Johnson expressed pride in the DoE SBIR program. Despite a small staff and limited financial resources, he said it had been able to provide:

  • On-time service: The Small Business Administration’s Policy Directive required that all Phase I grants begin within 6 months of the solicitation’s closing date. DoE had achieved that requirement for 20 consecutive years.

  • A modest lag time between Phase I and Phase II: The department kept this gap to no more than three months, providing continuity for awardees. When the lag time is longer, businesses have difficulty holding their teams together.

Need for Additional Resources

He said that with additional resources, the department could do more, including:

  • Outreach: Current resources limited the DoE’s outreach staff to three people, who were able to make public presentations only at the two National SBIR conferences and a few regional conferences each year. They had to turn down invitations from many states to address potential applicants about the SBIR program. Additional resources would allow DoE to alert many more small businesses of the advantages of participating in SBIR, especially in rural areas and states of relatively little commercial activity.

  • Phase III follow-up: The department’s knowledge of Phase III success was limited. With more resources it could study more cases and share their stories with potential small business participants, potential investors, and members of Congress.

  • Responsiveness: The DoE would have increased ability to respond to inquiries and problems from small business applicants and awardees.

Dr. Johnson concluded by recommending that the study panel examine the possible benefits of allowing agencies to use a fraction of the set-aside (he suggested 1 percent) for administrative expenses, thereby increasing the ability of the SBIR staffs to provide better service to small businesses.

DISCUSSANTS

Rosalie Ruegg

TIA Consulting

Ms. Ruegg began by saying that she was “one of those rare people who thinks that evaluation is kind of nifty.” As a member of the study panel, she said it would be a good idea if the study itself was evaluated.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Thinking about How to Measure Success

She said the panel should think about how to measure “success” for the study, and how that measurement could best be used downstream. The evaluation would not be a success, she said, unless the panel could say that:

  • The SBIR is working better than it was previously;

  • It has overcome institutional barriers, to the extent they exist;

  • Best practices are in place;

  • SBIR projects were being tracked with respect to output measures;

  • There were ways to measure long-term impact, including:

    • Direct effects, in terms of companies funded;

    • More indirect effects, such as others using the knowledge disseminated from the projects, the capacity that firms develop internally to do more, and beneficial networking effects.

“We the evaluators need to apply the same standard to ourselves that we apply to others,” she said.

Progress Through Conflict

Before starting her company, she had led the evaluation of the Advanced Technology Program during the period 1990 to 2000. The job was a “hot seat,” she said, because of the pressures on the program and the demand from applicant companies, and yet it was “very good schooling at the intersection of science and technology, public policy, ideology, and empirical evaluation.” These various forces, she said, “conflict, enrich each other, and bump off each other, and at the end of the day I think there’s a lot of progress made.”

Evaluation as a Management Tool

She then said that evaluation has its limits. “I love it, but it can be misused, so a lot of care needs to be brought to the application of the evaluation.” She said she felt optimistic about having this SBIR study done through the National Research Council, because it provided an excellent forum for organizing and implementing a “fair, balanced and constructive” evaluation. She said it was important in carrying out the evaluation to look not only at results but also at the process of the evaluation itself. She urged that during the study the panel members encourage organizations to make evaluation a “daily kind of process—not just to generate a report, but as a management tool, to improve operations.”

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Assessing Both Direct and Indirect Effects

She said that the symposium had just heard that DoE had very rich databases, which meant it would be “a pleasure working with them.” She urged that the study panel consider the “expectations of the study, which are downstream,” and the assessment of both direct and indirect effects, “where we look beyond the companies to see how others are using the technology developed.” It is useful to know as much as possible not only about the agency as it pursues its mission, she said, but also about others who are “out several steps” in the process of transferring technology to the marketplace. Too often, she said, an SBIR evaluation stops with the individual company. But it is also interesting to “look beyond that, to see who’s using the processes or products that come out of it: who’s using the journal articles that might be written, who’s using the patents, who’s citing the patents, how are the patents being used, how are the companies enriched in terms of doing other work.”

Attempts to Measure “Elusive Effects” of ATP

She said it was probably not possible to create a complete picture or measure of innovation in the SBIR program, but cited three previous attempts to describe some “elusive effects” of research for the Advanced Technology Program (ATP) in the Department of Commerce. She said that while these impacts appear “rather concrete” to most economists, they are viewed as abstract, if not esoteric, by many non-economists, who view only private returns as “real.”

  1. The “halo effect”: Receiving a federal grant may increase the ability of a firm to attract additional funding—the so-called halo effect. Early attempts to quantify the halo effect consisted largely of crude efforts to ask ATP participants whether they experienced increased success in finding additional funding as a result of getting an ATP award; most said they had. A more recent study19 approached this issue with more rigor, controlling for a number of other factors, and also using a control group of non-winners for comparison. Applying econometric techniques to survey data, the authors concluded that receipt of one ATP award indeed produced a halo effect.

  2. Knowledge spillovers: Knowledge spillovers are an important social benefit of research, but they are indirect and difficult to observe or measure. One way early research addressed knowledge spillovers was to count patents and patent citations. Three researchers at the National Institute of

19  

Maryann Feldman and Maryellen Kelley, “Winning an Award from the Advanced Technology Program: Pursuing R&D Strategies in the Public Interest and Benefiting from a Halo Effect,” NISTIR 6577, National Institute of Standards and Technology, 2001.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

Standards and Technology20 developed a new model based on fuzzy logic and network analysis. This analysis took into account the extent to which those citing patents are connected to or embedded in the larger national innovation framework of universities, other companies, and national labs. With their technique, some citations were considered to have greater value than others. This work appeared to improve the ability to assess knowledge spillovers by using a given project’s patent output.

  1. Market spillovers: While the direct gains of innovating companies can be quantified, it has proven difficult to measure more general consumer welfare gains (also referred to as consumer surplus or market spillovers), another important social benefit. Molly Macauley and David Austin, economists at Resources for the Future (RFF), developed a new index method that built on previous work for estimating consumer welfare gains from new technologies with certain characteristics.21 Their technique had the advantage of requiring less data than earlier methods.

Ms. Ruegg concluded that these three examples illustrated the progress being made in measuring effects which seem abstract and difficult to quantify. These effects are critically important components of social benefits because they underpin the rationale for public support of research. “We cannot hope to do a comprehensive measure of all of these effects,” she said. “But I would be quite positive about capturing snapshots and indicators of the presence of elusive effects.”

David B. Audretsch

Indiana University

Dr. Audretsch mused that this symposium was turning one of the old adages of science on its head. He had always heard that innovation is “all about standing on the shoulders of giants.” Today, he said, he had heard that “innovation in the SBIR is all about standing on the shoulders of midgets.” He said he had also learned that the SBIR had evolved from being largely ignored in the early 80s, both by academics and the media, to become the “sleeping giant of American economic policy.”

20  

Michael Fogarty, Amit Sinha, and Adam Jaffe, “ATP and the U.S. Innovation System–A Methodology for Identifying Enabling R&D Spillover Networks with Applications to Microelectro-Mechanical Systems (MEMS) and Optical Recording,” NIST GCR, National Institute of Standards and Technology, forthcoming.

21  

David Austin and Molly Macauley, “Estimating Future Consumer Benefits from ATP-funded Innovation: The Case of Digital Data Storage,” NIST GCR 00-790, National Institute of Standards and Technology, 2000.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
The Evolution of Evaluation

Just as the SBIR had changed, so had the evaluation of the SBIR. He noted Dr. Johnson’s insights into how evaluation had matured from some of the early studies by the General Accounting Office to the more complex and ambitious ideas expressed by DoE and by members of the study panel. For example, he said, the pioneer studies, when the SBIR was less well known, focused on the DoD, largely because it was the biggest SBIR agency and because it did extensive internal studies. Most of these exercises judged progress on the basis of only three somewhat restricted points: (1) had the program achieved certain straightforward benchmarks, (2) had the SBIR promoted the agency’s mission, and (3) was the firm itself strengthened during its involvement with the program.

He noted that Dr. Johnson implied that these were good starting points, but that “it’s clear from what we heard this morning that the mandate is to generate an innovative economy and innovative small firms.” It was not as important that each firm receiving an SBIR award be innovative. He again cited a point by Dr. Johnson that an overly narrow focus on individual SBIR projects might miss larger benefits or effects that develop indirectly or emerge as some related product or service. “I think that’s a very important insight for the evaluation team,” he said. ”If we make the metric too narrow, we’ll miss some of the benefits.”

Searching for Spillover Effects

He noted a challenge for the study panel: that if some benefits spilled over from the SBIR project to other products and services of the firm, it might also be true that extra spillover value goes outside the firm. He told of interviewing an SBIR firm in Indianapolis, founded by a university medical researcher who had developed a technique to produce obese male rats for diabetes research. He asked the researcher about the impact of his first SBIR award. The researcher said that the program had been excellent, but the product had not been commercialized. However, he had then applied for a second SBIR award based on a new idea—producing aged rats for other kinds of research. That idea had come from another firm that had gone out of business when the founder accepted a position at another university. The colleague at Indianapolis had realized it was valuable, decided to pursue it, and advanced that new idea with the help of the SBIR program. Obviously, said Dr. Audretsch, evaluating the “success” of the original company would not have been easy using traditional metrics. “I had gone to talk to him about one SBIR award that had produced nothing, but he was actually building on another SBIR award that had originally gone to another firm.”

The Real Goal: An Innovative, Competitive Economy

He said that the concept of firm evaluation was evolving into something more sophisticated, “outside the narrow units of observation.” It was important to

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

know that firms actually receiving the SBIRs were successful, he said, but the real goal was to create an innovative, competitive economy. The challenge for the evaluators, he said, was to identify connections between cause and effect that are often elusive.

He suggested that this challenge was going to push the evaluation process into more areas than earlier evaluations. He seconded Dr. Feldman’s plea for more data, and Dr. Johnson’s suggestion that access to managers in the agencies would be invaluable, so that the study panel “can actually talk to the managers and learn from them what are their perceptions and how the SBIR program has been integrated into the creative goals of the agency’s mission.” The panel should also be aware, he said, of how the program can sometimes change the career trajectories of small-business scientists and engineers.

Widening the Reach of SBIR

He noted that the goal of stimulating a more innovative economy was especially important in places like Indianapolis, where there was “a slight innovation deficit, a sense that people cannot or do not know how to” start their own companies. He recalled asking the medical researcher if he could have started his company without the SBIR. The man had said he had tried to secure a loan from a bank. This had failed because he had been unable to tell them exactly what he was going to produce, had no experience running a company, and had no assets other than the knowledge in his head. Fortunately he had heard about the SBIR from a colleague at the university, and, equally fortunately, the state of Indiana welcomed SBIR programs, even though it had few of them.

Dr. Audretsch described the job of a colleague whose job was talk to scientists about potential projects at federal agencies. He suggested that the study team might look at what were “the actual roles of states and local partners” and how such partnerships could be leveraged to make the programs more effective.

In conclusion, he said that the study panel might gain a fuller sense of the value of the SBIR by looking more broadly at its effects. “It’s not just about the projects that get funded,” he said, “or even the firms that get funded. It’s about changing the American economy to make it more innovative than it would have been otherwise.”

DISCUSSION

Technical Assistance to States

Patty Forbes of the Senate Committee on Small Business and Entrepreneurship said that reauthorization of SBIR in 2000 allowed the program to offer states and parts of states more technical assistance in applying. This effort was administered through the Small Business Administration, and she hoped that “by the time SBIR is reauthorized next time we’ll have some sort of data on whether it helps.”

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Gauging Program Success

David Finifter of the College of William and Mary asked two questions of Dr. Johnson: (1) What were the dates of the report he referred to, and is it available outside the agency; and (2) could he describe the elements of Phase III.

Robert Berger, SBIR Program Manager for the Department of Energy, answered these questions for Dr. Johnson. He said that DoE’s definition of Phase III was “rather simple,” with at least two indicators of success: one was sales, and the other was additional investment in the project. Within each of those, he said, DoE was interested in whether the money came from the private sector or from the federal government.

He added that he agreed with everything Dr. Audretsch had said about trying to find additional SBIR benefits beyond these two indicators, but cautioned that the task would be difficult. He said that DoE had tried to ask companies about individual projects, including the sales generated, but that the “company would make up numbers. They did not track things that way. They’d track sales by their own products and services.” DoE then asked the companies what were the products and services derived from all their SBIR projects, and what sales and additional development funding could be attributed to those products and services. “They could at least measure that. Only then did we ask, which of our projects contributed to those products and services. We tried to get to it from the back door.” He added that this information had not been issued publicly, but was internal DoE data last analyzed around 1998.

The Role of the National Laboratories

Jacques Gansler asked about the role of the national labs in the overall DoE program. Robert Berger said that the labs had no role in the SBIR program, except as potential contractors of the DoE. The labs, he pointed out, are not part of the federal government, but may be subcontractors on SBIR grants if a small business wanted to use them. A small business might also use a university or a large business as a subcontractor. National labs were available as partners on projects of any agency, as far as the DoE is concerned.

Use of Firm Data

Richard Coryell of the National Science Foundation, one of the innovators of the SBIR program, asked Dr. Berger about the availability of data from the agencies that had been obtained from private firms under conditions of confidentiality. Would such data be useable in the evaluation if the company was identified?

Robert Berger said that if he received a request for such data, he would ask the advice of their legal counsel.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Balancing Technical Merit and Commercial Potential

Rosalie Ruegg asked Dr. Berger if he had noticed any difference in the way the agency weighed projects thought to have good commercialization potential versus those thought to have high research value for the lab. She said that in the research areas mentioned by Dr. Johnson, such as fossil fuel, energy efficiency, and renewable energy, emphasis was placed on whether topics furthered the research goals of DoE. Was there also emphasis on commercialization potential, she asked? And how did the emphasis vary within the agency?

Dr. Berger said that DoE used the same criteria across all its offices. In both Phases I and II, there were three criteria: (1) scientific and technical quality, (2) was the company qualified to do the work, and (3) what would be the likely impact of the project. Under (3), the agency might be interested in both technical benefits and economic benefits. In addition, in Phase II, the DoE took literally the contents of Public Law 102-564, which instructed SBIR agencies to consider certain factors related to commercialization: 22

  • The company’s track record in commercializing previous SBIRs;

  • Whether the company was willing to share costs during Phase II, which indicated a strong company belief in its project;

  • Whether the company had commitments for funding at the conclusion of the SBIR grant.

DoE considered all of these factors, and also asked whether there was a difference among the programs, since some programs might relax one criterion or another for various reasons. “We demand very high scores,” said Dr. Berger. “We don’t want there to be doubt about any one of these three criteria. At the same time, one doesn’t have to be fully excellent on all three.” He noted that in some objectives of the department, such as the desire to move new hybrid electric vehicles into the private sector, SBIR might relax the research criteria. In a program in high-energy and nuclear physics, by contrast, the program might decide that the commercialization payoff is less important than the science.

He noted that companies themselves did not care whether the programs relaxed one or another selection criterion. The companies were more interested in obtaining rights to their technology in order to commercialize it. He said that DoE certainly saw the potential for high-energy physics companies to go to the private sector with products developed out of SBIR.

22  

Small Business Technology Transfer (STTR) Program 1992 Public Law 102-564, S.2941.

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Collaborating with National Laboratories

Charles Kolb of Aerodyne Research had two questions. First, he said that his company and companies he met with regularly felt the SBIR solicitation had begun to place stronger emphasis on the availability of the national laboratories as SBIR collaborators. He said that some people saw this as an opportunity for another strong private-public collaboration in science and technology, but that others saw it as a possibly cynical attempt to redirect some of the SBIR set-aside money back toward the national laboratories. He asked whether more Phase I and Phase II programs with national laboratory collaborators were being funded, and whether having a national laboratory as a collaborator gave a company a competitive advantage.

The second question regarded successful SBIR projects his company had had with national laboratory partners. He had found that the laboratories had different ways of interacting with small businesses. Some insisted on implementation of a full CRADA before they would agree to a partnership, as well as full payment in advance for their participation. Other laboratories behaved more like university or private-sector contractors. He asked whether the department had tried to set a framework of best practices for the labs in interacting with SBIR companies.

Robert Berger said that there had never been any attempt to coerce small companies to use the national laboratories as partners. In all the listings of available national laboratory facilities, only about 20 of the DoE SBIR projects had any collaboration with a national laboratory. He said there was more collaboration with universities than with national laboratories.

One of the reasons why national laboratories had been mentioned in the solicitations, he said, was that the program had begun to place solicitations for both SBIR and STTR projects in the same document. This was because the number of applications to STTR had fallen to a low level. Rather than fund poor-quality projects, the department decided to combine solicitations so that properly-positioned companies could apply to both programs at once, increasing their chances of winning an award.

In regard to the occasional difficulty experienced by small businesses in working with national laboratories, he said that his office was trying to respond to each complaint and had asked the national laboratories to “try to grease the skids. You do not want to have a small business in Phase I spend 3 months of the Phase I project trying to get their CRADA done with the national laboratory. This places the company at a disadvantage for Phase II, because Phase II is competitive, and the agency insists that all Phase II applications come in at the same time.”

Opportunities and Challenges in SBIR

Charles Wessner asked the DoE representatives what they liked best about the SBIR program, and what they viewed as sources of difficulty from a manage-

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

ment perspective. He also asked Ms. Ruegg if she thought it would be possible to capture the “elusive measures of outcomes” that Dr. Audretsch had proposed.

Dr. Johnson said it was easy for him to say what he liked most about the SBIR program. His father had come from a small business background and had encouraged him in many ways to experiment and design new equipment. But his father never went to the Small Business Administration for loans because the application process was too cumbersome. Instead, he simply used his own money to start his own company. Dr. Johnson said that the best part of the SBIR program was that it provided a simple way to encourage and fund the people who had good ideas. “Then it’s up to them how far they can go.” The same was true for the Office of Science, he said, which “lives and dies by the quality of the science we produce. If we produce lousy science, soon we won’t get much money for it.” He wanted to use part of the SBIR money to create more opportunities for people and to “reach out” in a way that raises interest in science. “Some day we may have a Nobel Prize winner who came from a background of a project like this.”

As for what he liked least about the program, he said that he wished he had more money to put into it. With more resources, the staff could do better outreach with the states, work with more recipients, and help them benefit more from their SBIR funding.

Robert Berger added that as operations manager he had a slightly different perspective. Most of his job dealt with the logistics of evaluations and solicitations. But occasionally, he said, especially in the context of the Commercialization Assistance Program, he worked directly with the small businesses themselves. When he did that, he said that “it sometimes brings tears to your eyes—the enthusiasm of the people, and the difference they tell you this program makes in terms of where their company started and what it has grown to; what their vision is and what is possible; and just how good these people are at science and how well they understand the contribution they want to make to the economy.”

He said that if he had to search for something he did not like about the program, it was having to rely on a small, overworked staff to produce the solicitations and evaluate the grants. “Most of them are very enthusiastic about SBIR and the companies they work with,” he said. “It’s those that are not that disappoint me.”

Rosalie Ruegg, answering the second part of Dr. Wessner’s question, addressed the difficulty of measuring the impact of research-based projects. She said that, to be sure, the study panel would look first at the easier, more quantitative kinds of metrics, such as sales, patents, publications, and commercialization of products. But she also repeated her earlier point that she was “pretty encouraged about addressing at least pieces of the more elusive effects.”

Administering SBIR

Robert Kispert of the Massachusetts Technology Collaborative asked why special administrative resources were required to support the program, and

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×

whether it was more expensive or difficult for agencies to administer than other R&D programs.

Robert Berger said that SBIR proposals were less efficient to review, per dollar of research funded, because of the large number of proposals received and the tightness of deadlines. While DoE had reduced the number of proposals by narrowing its topics, the proposal-to-award ratio was still considerably higher for SBIR than for other programs. Also, the performers were different, requiring more outreach than grantees at universities, for example, where the process of grant application is better understood. Finally, the agency’s peer review system was labor-intensive; DoE had to convey information packages to at least three reviewers for every proposal and retrieve then on time, or find substitute reviewers.

Jim Turner concluded the discussion by noting that the initial SBIR legislation benefited from the combined perspectives of the House Science Committee, the Senate Committee on Small Business and Entrepreneurship, and other partners in Congress. He emphasized that having representation from both small and large states and from both major parties created “a much stronger product.”

Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 108
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 109
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 110
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 111
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 112
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 113
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 114
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 115
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 116
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 117
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 118
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 119
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 120
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 121
Suggested Citation:"Panel IV — SBIR at the Department of Energy." National Research Council. 2004. SBIR Program Diversity and Assessment Challenges: Report of a Symposium. Washington, DC: The National Academies Press. doi: 10.17226/11082.
×
Page 122
Next: Panel V — SBIR at the National Science Foundation »
SBIR Program Diversity and Assessment Challenges: Report of a Symposium Get This Book
×
Buy Paperback | $61.00 Buy Ebook | $48.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

In response to a Congressional mandate, the National Research Council conducted a review of the Small Business Innovation Research Program (SBIR) at the five federal agencies with SBIR programs with budgets in excess of $100 million (DOD, NIH, NASA, DOE, and NSF). The project was designed to answer questions of program operation and effectiveness, including the quality of the research projects being conducted under the SBIR program, the commercialization of the research, and the program's contribution to accomplishing agency missions. The first in a series to be published in response to the Congressional request, this report summarizes the presentations at a symposium convened at the beginning of the project. The report provides a comprehensive overview of the SBIR program’s operations at the five agencies responsible for 96 percent of the program’s operations.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!