National Institute of Standards and Technology
Good morning, and thank you all for being here. I would particularly like to thank the National Research Council and Chuck Wessner for arranging this symposium and putting together a strong program on relatively short notice. We appreciate it.
We measure things at NIST. We have a favorite quotation—it shows up sporadically on bulletin boards and in footnotes—attributed to William, Lord Kelvin. He said that—I will paraphrase freely—when you can measure something and put some numbers to it, then you know something about it, and if you cannot, your understanding of it is of a ''meager and unsatisfactory kind.''
Since 1901 that statement or something much like it has been a touchstone for us. We are dissatisfied with things we cannot measure, and we work hard to remedy those unhappy situations. Obviously, it is the philosophical basis for the NIST Measurement and Standards Laboratories, the research core of our agency. It plays a strong role as well in the activities of our Manufacturing Extension Partnership and the Baldrige National Quality Program.
So of course when we received the assignment to create and manage the Advanced Technology Program (ATP), one of the first thoughts was, "Fine. How do we measure it?" We knew several things before undertaking this task:
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 31
--> Opening Remarks Ray Kammer National Institute of Standards and Technology Good morning, and thank you all for being here. I would particularly like to thank the National Research Council and Chuck Wessner for arranging this symposium and putting together a strong program on relatively short notice. We appreciate it. Importance of Measurement We measure things at NIST. We have a favorite quotation—it shows up sporadically on bulletin boards and in footnotes—attributed to William, Lord Kelvin. He said that—I will paraphrase freely—when you can measure something and put some numbers to it, then you know something about it, and if you cannot, your understanding of it is of a ''meager and unsatisfactory kind.'' Since 1901 that statement or something much like it has been a touchstone for us. We are dissatisfied with things we cannot measure, and we work hard to remedy those unhappy situations. Obviously, it is the philosophical basis for the NIST Measurement and Standards Laboratories, the research core of our agency. It plays a strong role as well in the activities of our Manufacturing Extension Partnership and the Baldrige National Quality Program. So of course when we received the assignment to create and manage the Advanced Technology Program (ATP), one of the first thoughts was, "Fine. How do we measure it?" We knew several things before undertaking this task: The ATP presents a unique measurement challenge, not just for NIST but for government, industry, and the economics community. We know that technological innovation, by and large, is a good thing.
OCR for page 31
--> Advanced technology has sustained our leadership in the global community. It has accounted for over 50 percent of U.S. economic growth since the Second World War. The Advanced Technology Program was established by Congress to help spur this economic growth. Congress created a mechanism to encourage and nourish economically valuable technologies that might otherwise go untried—or be exploited by our global competitors—because the R&D risks were too high to be supported entirely by the private sector. The ATP practices a sort of economic jujitsu, applying relatively small, highly leveraged investments against the potential for major economic impacts. Can it work? Well, sure. The classic existence proof is the Internet—the relatively modest early investments by the Department of Defense (DoD) and universities are utterly dwarfed by today's economic impact. Yet this raises another set of questions: How do you measure the worth of technological innovation? How do you measure its impact on the economy of a nation? How do you predict that impact five years out? Ten years out? Ten years ago few people would have been sufficiently insane to predict the current role that the Internet plays in the national economy. The ATP measurement challenge emanates from the program's mission to create long-range, broad-based economic benefits. Long-range means sometime out in the future. Although, ultimately, we will perform retrospective analyses, we cannot wait until then. Broad-based economic benefits mean tracking and measuring impacts that go well beyond the individual company or companies that did the original work. We need to measure or credibly estimate the macroeconomic effects of a fundamentally micro-economic program. Finally, we need to make credible predictions based on those measurements and estimates. Background on Program Assessment So how do we assess our work in the intervening time between the short and the long-term? At NIST, measurement and evaluation have been part of the ATP from the beginning. The first major report appeared in 1993, a survey examining the early impacts on participating companies of the first 11 ATP projects after a year of work. The ATP established its Economic Assessment Office to oversee a wide range of data-gathering and analysis efforts including basic data collection, case studies, surveys, and macroeconomic modeling. The ATP's business reporting system, for example, uses an innovative electronic survey form to gather quarterly information on active projects and follow-up data after completion, tracking their progress toward achieving business and economic goals.
OCR for page 31
--> Our research universe encompasses the 431 projects selected by the ATP for funding to date. We can tell a lot about the ATP just from the project profiles: Industry commits significant resources to these endeavors, making it a true partnership. The ATP committed to $1.386 billion in funding for the projects selected to date, less than half the projected $2.783 billion budget for the planned research. Diversified joint ventures play an important role in the program. These 431 projects involve more than 1,000 participating organizations, including more than 125 universities and nearly 20 national laboratories. Small business competes effectively in the merit-based selection process. More than half of the projects are led by small businesses. Surveys and the business reporting system tell us more about the ATP's near-term impact on participants: The program is stimulating collaboration, encouraging more and more research joint ventures. The ATP is accelerating the development of high-risk technologies. Eighty-six percent of project participants in one study said that they were ahead in their R&D cycle after just one to two years of ATP funding. Thirty-nine percent of those were ahead because they would not have undertaken the project at all without the ATP. ATP is fostering the development of leapfrog technologies that go well beyond simple incremental advances in the state of the art. Thirty-seven percent of ATP-developed technologies in a recent study were "new-to-the-world" innovations. These technologies have broad potential applications. Companies report identifying an average of 4.5 applications per project. Grant recipients have committed their own resources to commercialize more than 100 new products, processes, or services based on ATP technologies. Recent Assessment of the Program's Economic Benefits The ATP is probably the most thoroughly studied program of its kind in the world. My current stack of reports, studies, analyses, and analyses of analyses on the ATP is about 9.5 centimeters deep, or about a centimeter or so of report for every year of the program's existence. I am not sure the collection is complete. In addition to near-term results, we have applied macroeconomic models in an attempt to look at potential long-range impacts. In 1996, CONSAD Research
OCR for page 31
--> Corporation used a 53-sector macroeconomic model developed by Regional Economic Models, Inc., to estimate potential long-range impacts of a single ATP project. This was a joint venture under the Auto Body Consortium, a coalition of small-and medium-sized auto industry suppliers working with Chrysler and GM as well as the University of Michigan and Wayne State University. The project developed a suite of process monitoring and control technologies to reduce dimensional variations—fit and finish problems—in auto bodies on the assembly line. These technologies led directly to lower production costs, lower product maintenance costs, improved product quality, and reduced cycle time to launch new models. On the basis of their modeling, CONSAD estimated the total impact of this project on the U.S. economy—counting all of the industrial sectors affected by the auto industry—at more than $3 billion in the year 2000. Last year the Research Triangle Institute (RTI) Center for Economics Research completed a study that was intended to develop a prototype methodology for predicting the long-term social benefits of public investment in technology development. RTI analyzed probable partial outcomes for seven ATP projects in the field of tissue engineering. These were technologies to make "template" prostheses derived from biomaterials to assist the body in rebuilding lost tissue, to produce implantable cells that generate key human hormones and other bioactive agents, such as insulin for diabetics, and others. RTI used a procedure to calculate the expected social return on public investment—the extent to which the nation is better off as a result of the ATP. This includes not only the private return to the innovating company, but also the value of the technology to the nation at large—lower health care costs, for example, or improvements in medical outcomes and quality of life for the patients. The ATP expects to invest a total of $15.5 million in the seven projects covered in the RTI study. RTI calculated the expected social return on the ATP's investment in these technologies at $34 billion in net present value over a 20-year period. That is a predicted social rate of return on the ATP investment of 115 percent annually over 20 years. This is not a bad return. You might be interested to know that in both cases we urged the researchers to use conservative assumptions. Dedicated optimists could come up with even higher figures. I have asked ATP officials to be careful about touting these results because they are so striking. I cite these results not because I necessarily expect you to accept them at face value. Rather I want to emphasize the magnitude, and the difficulty, of the task we face. There are major issues involved in the use of macroeconomic models on
OCR for page 31
--> a program of this size of the ATP, but we need to ask these predictive questions. These reports represent the sort of analyses we need to have. Assessing Completed Projects Today we are adding a new report to the stack, a summary study by economist William F. Long, called "Performance of Completed Projects," which examines the first set of completed ATP projects.3 This is the first comprehensive review of the outcomes to date of the ATP portfolio, and the first of a series of studies following up on the results of ATP investments. Dr. Long's report covers the 38 projects that were completed as of the end of March 1997, documenting research accomplishments, subsequent work by the companies to commercialize the results, and near-term outlooks for the successful technologies. It also looks at 12 projects that were terminated or canceled during that same period (out of 280 projects selected). This is a respectable rate for a program that concentrates on high-risk research, and we have learned some useful things from the failures. It is, as Secretary of Commerce William Daley said in his announcement, "a portrait of a program that works." The program's current portfolio covers the high-technology terrain very thoroughly. For example, there is a process that merges tissue engineering and textile weaving to help regenerate lost or damaged tissue in the body. There is an application of high-temperature superconductors to improve cellular phone service. We are funding a desktop bioreactor capable of growing large amounts of human stem cells isolated from bone marrow for cell replacement therapy—now in clinical trials. Already in commercial use is an ATP-funded computer programming tool to simplify the task of writing software for parallel-processing computers. Finally, there is a novel technology for processing very large semiconductor wafers. This innovation, developed by Diamond Semiconductor Group, has made the United States the first in the market with processing equipment for the next generation of 300-mm semiconductor wafers. These are just some of technologies that the nation has, here and now, because of the ATP. As the ATP enters its tenth year, we face some significant challenges. At the same time, the ATP, as Secretary Daley says, is working. We are, without a doubt, having an impact. Dr. Long's report is only the most recent testament to that. However, we need to grow the ATP to a level consistent with the program's fundamental mission of economic growth for the nation. Just what that level should be is a matter for debate, but as an example, I would point to the only other major technology funding program with demonstrable success and national impact. That is the Defense Advanced Research Projects Agency, with a budget in the neighborhood of $2 billion. 3 William F. Long, Advanced Technology Program: Performance of Completed Projects. Status Report Number 1, U.S. Department of Commerce: NIST Special Publication 950–1, March, 1999.
OCR for page 31
--> To support this, we need the data. The challenge is to document how the ATP affects the economy, and by how much. The challenge is to do so in a rigorous and credible way that will satisfy the requirements of the public policy makers in Congress and the administration. Our goal in this workshop is to address those measurement and assessment issues. How can we best assess the ATP? What data do we need to do that? How can we get the data we need? I will leave you to it. We appreciate your help and look forward to your insights. Thank you.