Papers Commissioned for a Workshop on the Federal Role in Research and Development (1985)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

ECONOMIC M~SL=ES OF ME REMANS TO FEDERAL RESEARCH ~~D DEVELOP.MENT Peter C. Reiss* Graduate S choo 1 o f Bus ine s s S tanford Unifiers i By This pacer surveys the retool ogles es Lima he the re turns to fed era: ID . cor~cep dual issues and caiculac~ng the contra buttons to and coses of federal R&D programs, The second part surveys the mecI2odo loges that economists use co measure The retrurz2s co federal R&D. :r a iso reviews severe: appl~-arions of chose techniques. The conclus~o-. assesses whar we are likely and nor likely co learn from existing tec,~i ques and] dare . cha ~ economic es use So The f irst Parr ouch ines measurement orobl ems associated hi cI2 Research and development (Rho) poLic~akers have Tong been interested in ache question of whether it is possible to measure accurately the relearns to federally financed.R&DO This interest is fos~cered not only by national policy concur Its of what private R&I) activities should be sponsored or subsidized, but also by issues of how incremental funds should be allocated among publicly managed proj acts . Recent accelerations in the size and scope of many federal R&D programs, mounting federal deficits, and ~ heightened perception that the United States is toeing technological advantages at home and abroad have reinforced economists' and polic~alcers' demands for more precise measures or forecasts of the contributions made by federal R&D. This paper revsews the ~cechniques that economists use to provide operational measures of ache returns to federally funded R&D programs . ~ comprehensive survey of all the methodologies used to calculate returns deco different federal R&D progr~m.C is beyond the scope of this paper. Ins~cead, the paper reviews the ways in which economists measure the returns to direct federal outlays for R&D. For the most part, the paper con~crasts economists ' notions of how returns to federal R&D could be measured with what actually has been attempted. * The author is graceful deco Ellen Pint for research assistance. - 87

It should be emphasized at the outset that there is by. no beans complete agreement among economists on the maj or methodological or measurement issues associated with those calcutat'ons. indeed, some economists have questioned whether the exercise of calculating returns is at all useful given our ~ imi-ed 'knowledge about federal Red) spending programs. The paper therefore concludes with some observations on the limitations of rate of return calculations gi hen present conceptual ambiguities and data problems. ECONOMIC ISSUES AND MODELS Many studies have calculated -he contribut ons of private A&) . These studies have been reviewed and critiqued in a series ot pacers, inc. lu ding7Ggi l i ches, 2 ' ' ' 4 bans f i e Id, Cans f is id .e t al, and Tertec~rj '~ . To the extent that one dollar of federal R&I) is s imilar eo one dollar of pr: crate R&D in its effect on economic growth and technical progress, this sec than will repeat Cal contained in those surveys. There are, however important differences between federal and private Rag) inputs and the types o f technical change ~he'; are likely to produce. These differences have received relatively little attention from economists ~ argely because of the difficult problems associated with Identifying and frilling the consequences of federal R&8 in~restment~' (There are some exceptions. See Gr=tiches, Terieckyj, Levy and Terlecky~, ~ and the references cited therein. ~ Specific conceptual and measurement problems associated with federal R&D are discussed in detail below. Before discussing these problems however, the basic economic paradigms and terminology associated with the economics of R&D are reviewed. there are many economic models of the research and development process. ~2 The traditional economic model uses neoclassical economic theory and production func~cions to describe what R&D does . In the production functi on approach, R&D is viewed as an " input, " along witch capital, labor, and other inputs, into a process that generates ~outputs. n Often, inputs are taken to be such diverse factors as ideas, scientists, equipment, or expenditures. OUt:pUtS are less clearly defined quantities, such as changes in Product quality, technological progress, or productivity growth. A fundamental question that is not addressed easily by economic theory is: What are the relevant units for R&D inpu~cs and outputs? Traditionally, economists have tended to view R&D inputs as an aggregate quantity of physical factors and not a dollar amount (such as federal expenditures ~ . However, because they rarely have data on ache manpower and cap ital inputs used in the Ray process, economists proxy R&D inputs by deflating coccal R&D expenditures with crude input price deflators . t4 Economists ' measures of the output of the R&D process have traditionally been measured in a s imilar way: R&D inputs generate units of output, or units of quality improvement. Because reliable technological output data are often difficult to obtain, economists frequently use deflated sales data to measure — 88 —

r.i~s of output, and proxies such as patents to measure latent chno Logical change . Economists relate these observable or ~~nabservable input and output constructs to each other by specifying an R&D 'product on function. " Usually, chat production function is either a deterministic or a stochast~ c mathematical relationship between inputs and outputs. Economists add to the production function an economic decisionmaking model that predicts how much R&D firms wi li undertake . Economic models of the R&D process differ typ ically in what they assuage about economic decisionmak~ng. The neoclassical approach assumes that firms decide on R&D inputs by max:~tz~ng the expected profitability of their investments. From this assumption, both R&D demand specifications and output supply equac' ons can be deri led. Such equations relate prices for R&D inputs and outputs So the observed leered of R&D (see, for example, Goldberg ~ ~ . Other models use evolutionary or behavioral approaches ~ see Nelson and .~'interl6 ~ to derive predictions about how much R&D firms will undertake and what R&D outputs firms with produce. Although this description of economic models of R&D is highly stylized, it does emphasize the two maj or components of ~ modes of the R&D process: (1) the specification of what R&D does as an ~ noun and (2) a set of economic assumptions about how R&D is allocated by economic agents. It would seem that the first component is ali that is really necessary for calculating what federal R&D contributes to technical change. For example, one could calculate accounting rates of return to federal R&D once it was known how to raeasure the present value of inputs and outputs. 'However, ~t is essential Icy recognize that the second component is necessary for interpreting the measures constructed. that is, if economists are interested in policy questions such as, Are races of return high or Low?, or, Should they invest more in R&D?, then they need to recognize that races of return are determined endogenously by both market and nonmarke~c opportuni~ctes. Those opportunities themselves are influenced by technological change and the varying degree to which technological innovations confer readily appropriable returns Thus, for instance, deco determine the consequences of increasing defense contract R&D funds, one needs to understand how corporate incentives respond deco federal contract provisions. It may well be that the promise of increased federal support encourages companies to divert productive R&D resources to socially wasteful competitions for future governmen~c contracts O As will. be seen in the methods discussed below, such incentive issues often create subtle methodological problems, which economists have great difficulty in modeling. Finally, economists need to contrast what they would like to measure with what will ever be feasible. It is clear that some of the consequences of federal R&D will never be measurable. For example, the political benefits of a strong national defense or a sophisticated space program are difficult to reduce to numbers, as is — 89 —

the desirably ty of income red, stribucion caused by techn, Cal change. There also are some benefits that are potential: y quantifiable, but for 'which there are no adequate data at present. Me limited progress in quan~c~f ring health care i~npro~rements is a ready example of the data problems that exist. Economists are lef_ then to try and measure those contributions ~cha~c are readily identifiable and easily quantifiable. The usual yardsticks on which economists base their studies are rates of return. Rates of return are usually of two Herpes: those that distinguish between the priorate contribution of Rat) and those that measure the social contribution of Red. The term "private'' rate of return is used ~ o identify rewards deco the f in performing rise R&D . That re cum therefore quantifies how much of the R&D is appropriable through increased profits and sales . We term " social " rate of return to R&3 is used most often to summarize all the returns realized from technical change. Therefore. the social rate of return includes private returns as well as negative and positive spit lover effects of the R&}~. Lee difference between private and social rates of return is called the " external " race of return. tic measures ache added social value of R&D that private firms are not able to internalize. It is this rate of return that interests policymakers. Me paper now turns to a discussion of the different methods ~~. economists use for calculating the contribution of federal R&D Ho technical change. A subsequent section will discuss some of the difficulty es associated with constructing R&13 input and output measures . METHODS ~~ ~TIONALES FOR =TE OF RET CAL~TIONS Marring reviewed how economists think about the Ed&) process as a relationship between inputs and outputs, the paper now wit! address how economists use input and output data to make s tatements about the contributions of federal R&D o The most common techniques and the ir limitations are satirized in the sections that fo flow . Readers interested in ~ more complete interpretatian of the surveys discussed are refarr:' to "he original articles and to Nestor Teriecl~r~ ' s sugary, also presented at this conference. The main methods and topics covered in this section are productivity studies, complementarily analyses, discounted races of return, and surreys. PRODUCTIVITY ANALYSES Productivity analyses relate changes in ache growth rates of outputs Deco changes in ache growth rates o f inputs, inc. luding R&D . Thus, in the language o f the precarious section, this approach assumes that ache output of Rho inputs is growth in r.he output of goods and services. — 90 —

This methodology has been eased extensively co document ~ h:Oer~ect of federal A&) in agriculture ~ mining, and manufacturing . I ~ has not, however, been applied widely to high technology or service sectors such as health, space, and defense, because economists lack detailed information on the outputs of goods and servi ces in those sectors. For example, it would be difficult to use this approach to quantify how much CAT scarers have increased ache produezivi ~y o f doctors or the quality of patient care . I ~ is 2robab le that CAT scanners have affected mainly the q,'=li~cy and not the quantity of patient care, but there is disc file hard evidences to sugges ~ to what extent Chat is true. Lee methodology i Self is ~ argely econome~cr.c in orientation. ignores some of the individual deceit of specific obser',ations in favor of estimating a general con~cribu~cion of federal R&D to ei.he of two economic ~rariables: focal factor productivity or the aggregate output of goods and services . To imp Cement the technique, the economist specifies a mathematical relationship between A&) inputs ~ such as federal R&D) and a leered of technological accomplishment. Len, the hypothetical level of technological accomplishment is related to the level of physical outpu~c to produce a production function that links R&~) to the output or physical goods and services. A common specification is Q ~ A(K,~) F(C,L) (~) where Q - the Output of goods and services, a ° the technology level, BY ~ the stock of R&D (federal and private), T ~ a time trend, - r ~ the baseline production function, C ~ cap i Cal input, and L a labor input. Thus, Act, T) is the R&D production function discussed earlier, and R in the R&D input. Usually, the stock of R&D is specified as ~ weighted average of past levels of I&. The weights and the length of the average are either chosen arbitrarily (or limited by available data) or based upon estimates from other data sets (see Teriecicyj2 ). Two vexing problems that must be resolved in any estimate of K are how much con~cemporaneous R&D should be included in A, and should K be broken down into mul~ciple measures (for example, private versus federal R&D). Most studies Chat consider the effects of federal R&D usually include separate priva~ce and federal — 91 —

R&D stocks. Few studies, however, have attempted to disaggregate federal R&D into its various components: contract, public proj ec , and agency funding Rag). Thus, in practice, most studies assume than different types of federal ass istance have the same effect on output . The other panics of the production equation ( 1~ are fairly standard and need little explanation Time is i ncluded as a Variable is' A(. ~ to capture trends in output due to so-called disembodied technical charge. Me baseline production function, F. is a summary statement of how d~fferen~c combinations of capital and labor generate output in some base year (when A - ~ ~ . Several studies have estimated the contribution of federal R&D =o output by assuming that equation (1) has a Cobb-30ug: as form Q = a e~:h3C-ri-- whexe a ~ a constan~c numbers ant e ~ the exponential. faction. Ifs funct~or.ai form is convenient because it s Linear in its natural logarithms ~ lo) inQ = Era ~ aT 0 3lnX ~ , ins—~ I - Kiln ~ ~ 2) and linear in the time rate of change of productivity Q = ~ _~K +_cC -(i - ~> ~ . where there represents the time rate of change of -he variable. we linearity of bow specifications has contributed to their popularity as baseline models for gauging ache effects of Ran. Several studies in we late 19SO's and early let's estimated these production functions without R&D. Gril~ches24 was one of -the first ~co include federal RSD t~ such a spa :f.cation. Lat applications include Evenson, Minas~an, and Tern echo, hong o~chers. Me predation fus~c~c~on approach is not without its I:mita~cions, however. Not. that the functional form blurs the distinction between R&D as an input into A and R&D as an input into P. Moreover, R&i:) is neutral in its effects on capi~cal and labor complementarities. Finally, The abate production function presumes that production :alces place according to a constant-recu~ns- to Scale technology. Rarely is that assumption tested in practice, and it seems to be a tenuous assumption for sectors to which large amounts of federal R&D are supp lied . — 92 —

or Of ~en, ~ he abo~re direc~ app~ oach ~o es.i.mat ~,g =!~e produc-~on ~u~,c~ion meets with che criticism :;~at i ~ ignores .ne economic simultaneigy created by input and ou-?ut dec' signs (see, Gr' 1'ches ~ . This c~iticism led researc;ners ~o ?.efer procuctivit-y equations ~chat use economic assu`-np ~ions ~o es ._mace const.ained ~ime ra~e of change vers.ons of equ~tion (3~. :~.ese spec' -'cat~ ons kave the form ~. TFPG ~ ~ ~ ~ ~ r~ u .~, i ~ . G = ~—r _ ~nere (4) , ~ ~ ~ ) ~F=G - totat ~actor product~ vi-y g. owt5, R~ - net inves =ment in R&D ~ federal, private, or both), che elast~city of out?tst with respect ~co ;he R&D stocic, and ~ - a rate of return ~o R&~). The var~ aole ~ ocal ~ac~or produc~'ri~y is de-_ned as ou-2u~ di-~rided o oy an index of ~actor ~nputs ~ ~or ex~mpte, T" ~ ~ C- r~~ ~ Tota' =~ctor productivi~ growth, ~hen, is the time -ate or increase in T--= (TrEG ~ Q —~C —~ i - _l _ ~ . The coeff~cient ~ in equa~ioras (~) and ~ 5 ~ is ~he growth ra~e o disembodied ~echnicai change . The coeffic~ent B in equation represents how r such outpu~c changes in res~nse tO a sSe percent increase in ~he s~ocic of R&D . Gr:liches, E ~enson, and others ha~re obtained estimates of g between .03 and .07 for federal R&D in ag~iculturel a' though thei' es~imates var~- cor.siderably by3~;-e sample and ~he ~rpe of R&D being':nalyzede Stud, es by .~£=nsfieid, Ter'ec~;j, 3 Levy and Terleckyj ,' and others ha~re found B ~o be on the order of .07 ~o . 15 for ?rivate R&D, al~chough those es~imates als 0 vary dramatically from s cudy to s tudy . The ast~imate of B obtained from equation (4) is no-, strictly speaking, ~ ra~ce of recur`: to R&D. .~themat.cally, i" is rela~ed ~co a ra~ce of recurn by ~;~e fo~mula where the margir:al procuc~c or rate of return to ~he R&D s.oc~c. — 93 —

_conomis.s refer to the marg nai product of R&D, r, as a He of ~e turn to Rid). By est-=a~ing~equation (4), economists can recode- at. estimate of the rate of return to R&~ that Caries accordi`.g -o He output - to - cop i Cal rat ~ a . ~.~e ~ s where the ~ denotes an estimated value. On the o ocher hand, ~ f the ~ Hi. economist. esti.sates equation (5) because in ~ s easi er to measure ,? there art es ~i.'nate of the rate of return is obtained d~~ect' Y . However' Gnat estimate of r is constrained co be Dine same for al 1 oose~ions (projects, industries, etc. ~ in Me data set. (Sote that ~ n this case it is now the elast~c:~,~, 3, -~t varies across obser~rat~ons . ~ The choice of func Kiowas form, therefore, is mpor~cant because it malces assumptions about what factors are l. ke' Van to be constant in ache data. Estimates of - vary from application to application, as does i As interpretation as a return. In a recent study, Griliches and f icheenberg ~ (using a mode ~' cat on of the above equat' ons ~ estimated rates of return. for federals and private R&D simu, ~aneous~ ;. They found that the rate of return for federal R~ was at 3~0S.C ~ . 5 percent between 1959 and 1976. Private R&~, on the other hand appeared to have quite high rates of return ° between 9 . 2 and 33 ~ 4 percent. Griliches and L:chtenberg interpret those rates of return as es "imates of Cue "gross excess social rate of retuL`." to R&D . That terminology differs from earlier interpretations of roughly the same quantities and is in need of some explanation. ~~ is ~ social rate of return in that only the changes in Output ~ i US ted for the contra button of other inputs ~ are included. thus, ~ ~ does not include the value of private revenues or profits ~ hat arise because of ~echnologi-~1 change. Sote, also, that it is an incomplete social recur: because ~ t does not count all contributions of R&D. For example, it ignores consumer welfare. The return is a gross rerun because it does not net out all depreciation. F'r:all~r, i. is an excess returns in the sense that R&D is confounded. with con~ren~c~ona1 measures of capital and labor. 5 Although it is impossible in this snort paper to do i usable to all the effort that has gone into R&0 productivity studies, i- is worth noting some special problems and puzzles posed by federal 2&D for productivity analyses . Firs A, there is the issue of whether ache underlying economic production function approach amp l ies to 3 federal R&~) in the sue way that it does to private R&D. Mansfield arid o thers have observed that perhaps low rates of return to federal R&D are found because federal. R&D does not contribute directly to output growth. Ins Head, federal R&D may only enhance the profitability of pr-~ra.e Rap. lint hypothesis has yet to be explored fully. It would, for example, be interests ng to examine federal R&D contract data to consider to what extent contract R&D is not related direct _ 94 —

to applications ~ha: affect the output of ?ri~a~e goods and services -. the short run. Second, there are the accounting anj7ex?ens ing biases in rate of return caucus anions (see Scnanker:~an ~ . hose biases have not been explored in any detail for federal I&. ~h.'cc. e:;?ensed differently from private R&D. Third, there is =he policy' question of whether such estimates are at all usefu' for gauging federal R&D policies. To the extent that calculations such as equation ~ 5 ~ represent the effects of exogenously varied federal expenditures, they may be us efuL for allocating federal funds. However, there are good reasons to doubt whether es timbres of equation ~ 5 ~ satisfy these requirements . For example, federal support of semiconductor f" rms in the ~ 950' s and early i960' s influenced the growth of semiconductor output, if only through military purchases ~ see Lenin ~ ~ Further, the suggestion chat federal R&D could become ~ tool of industrial policy is heard own from advocates who urge, " Pick our winners and cut aid to our losers . " FED EARL R&D AS A COMPLIMENT OR A SUBSTITUTE The issue of whether federal R&D is a complement or a subs Boa private R&D has been debated actively n the ?°: ICY t'2 terature '~or some time. Economists hairy been interested in tests question also, because, to the epicene aches federal R&D increases priorate spending ~n civilian applications, the true contribution of federa1 R&D may be understated. Only recently, however, have there been many large ° scale attempts to gather detailed evidence on the question of whether federal R&D increases the amount of it&l) performed by corporations. Out of the statistical research on this issue has come a general impression that federal AND is ~ complement4,;5o privet: R&D efforts ~ see, for exa,m~te, Link, Levin and Reiss, Scott, ~ and Leroy and Terieclcy:- ). This general impression has4been reinforced by several cage s studies (see Mansfield, Mowery, and flowery and Rosenberg43 ~ . However, there also is evidence that noe ~11 federal R&D is a complement to private it&l) and that some private R&D ma~ be undertaken in an effort to obtain federal R&1:) ~ see L~chtenberg4 ~ . At present, there are few conceptual models of how federal R&1:) affects priorate Red) incentives ~ and, conversely, of how federal decisions respond to private ~nitiati4,s, such as those of defense contractors ~ . For example, as Mowery notes, if federal R&D is to be though~c of as increasing spillo~rers or lowering information costs, then federal R&D is only complementary if firms have the in-house R&D laboratories to take ad~rantage of federal assistance. Me firm's competi~cive environment also plays ~ role to the extent that competition affects the returns that firms anticipate from future R&D contracts. Another common argument suggesting that federal it&l) s simulates perorate R&D is that federal R&D reduces the risks firms face, thereby caus ing them to inures t more than they would in the — 95 —

absence of federal support. However, if 'hat is ~rue, then ?- =~~e _-rms have incentives to get the government to bear more of =he r i sk. In that sense, government R&D subsidies create free - ricer problems that lead to suos=~ution of Federal programs for ?r-~e . Me lack of a complete conceptual understanding of e~cact'y how federal R&;) complements private R&D has made it difficult tO8 formulate empirical tests for complementarities. Mansfield and his associates have done some survey work on this question ~ see the section titled, Survey .Yethods, below). hey asked firms ~ series o questions to evaluate Sow Such the firms would Caere spent on ene =- R&D in the absence of government support. Wey found, on the average, that only 3 percent of the private R&D performed w' Ah complementary federal assistance would have been conducted in _~.e absence of that federal support. .hioreo~rer, they found an ir~teres~~.g asymmetry in Ache complementarily relation. If the government were -o cut federal spending by 10 or more percents then for each one dollar decrease in federal energy it&l) spending, firms would decrease :;ne'~ spending by 69 cents over ~ period of several years. On the Ceder bared, if the government increased its R&D spending by ' O or more percent, funs would increase thesis R&D by 12 cents . =~c as Eve ~--~ has interesting implications for policy, but has vet to be explained or duplica~ced empirically. .H any econometric s curies of ache complementarily of federal and ornate R&D have used cross - section data to ex~n,~ne whether changes ire pr-vace spending are correlated with changes in federal. spending. Usually, the studies test for complementar~ty by es-~:~ating specifications of the form Drip ate R ED P., Tic RLD = a - f + ye Sales Bases where a. 3 anc - ~ parameters to be estimated' and ~ - a set of variables that influence pr_ fate spending. The coefficient B represents an estimate of the number of cents that a dollar of government R&D increases private R&D ~ as Ding sales remain constant). If it is positive, federal R&D is said to be a complement. If it is negative, federa2;gR~ is laid to be a substitute . Most estimates ~ see kink, Scott, Levin and Reiss, ~ and Terl eclcyj ~ are in the neighborhood of 7 to 10 3 cents and significantly different from zero . Levy arid Terlec'~j have, however, reported estimates as high as 2S to 27 cents for federal contract Red) (see also Char, es River Associates: ~ . Several i:~por~ant issues that the above work does not address seem fruitful areas for future research. First, =here is the ~ s sue — 96 —

c: whether corre cations of federal and pri bane R&D imply causaticr. . that i s, it is unclear whether federal R&D is al' ocared independency ~- o f fac moors that affect priorate R&D . Second, even if federal R&2 does affect private R&D exogenous ly, the estimates from the above specification Day not be very useful for policy purposes. For example, there is little evidence on its stabil't~r across sectors. f irm s ice, and so on. Also, whether these estimates represent marginal or average effects is not known. Finally, there is the issue of whether these numbers are "Large" and should be of concern to policymakers. For example, if it is assented that federal R&D influences total factor productivity growth only through ins efface on priorate A&) ( as the results of Griliches and Lichtenberg suggest), then the above n',rnbers ~ Imply char federal R&D increases the excess social rate of return to pri late R&I: by about only one percent . DISCOUNTED COST - BENEFIT METHODS Accounting rate of return methods are used often by corporations and polic ~ analysts to ca;gula~ce the profitability of R&D pro; ecus ~ see, for example, Horr~gren: and Ansani ~ lathe most popular methods for measuring ache ex post profitability of proj ects are discounted cash flow (present value or internal rate of return) and proj ect payback. The choice or technique depends tropically on the type of ?&D proj ect Hieing undertaken and the profit e of costs and benefits ~ see Ansani: ~ . Sometimes, the same techniques are used to assess the ex ante ~ expected) rat" of returns to atter`~ati~re proj ects . However, their usefulness as predictors of rates of return to R&D depends often on the ad; uswents made for the riskiness of proj acts and the time profile for future costs and benefits. The teas ic ingredients of accounting rate of return calculations are the present values of costs (Rho inputs and related expenses ~ and revenues (R&D outpu~c). The following section discusses many of the measuremen~c issues associated with calcula~cing costs and revenues correctly. Even if most of the measurement issues can be resoldered satisfactorily, there is a host of unresolved conceptual and computational issues associated with the calculations. For example, the calculations often include only the revenues of ire firm or federal agency that receives ache R&D (see Rubenstein and Terleckyj 50 ~ 61) . That results tn a "private" rate of return because it is limited to the costs and revenues of those who conduct the R&D . Such calculations are useful to ache f irms do ing the Red in that they indicate the profitability of the investment. They are also interesting to federal policymatcers and economists because they summarize private incentives and the opportunity costs associated with not doing R&D. Private rates of return, however, often do not present a complete pi cture of the returns to federal R&D or the desirabili~cy of private proj ects . The most important consideration missing from prig ate rate of return calculations is that they ignore — 97 _

externalities associated with innovations. Those externalities or s?illovers are indirect effects of the inno~rati be effort. They represent ache sometimes very subs~can~ci=1 gains (and tosses) chat c ons, me rs, c ompe t ing f i Ems Is and o the r s ec to rs o f tine ec onomY experience because of technological charge. Consider the following exhume of public and pri rate benefits . .Ass,tme that only two firms are competing for a massive federal R&D contract to design ~ new, fuel~efficient 3 et engine. Suppose that one of the firms gets the contrac~c, and invents a fantastically fuel-efficiene engine that has immediate commercial application; in ode interim, the other firm goes out of business. A private rat- o. return calculation for ache federal R&D (and any private R&i)) will probably indicate that this is a very successful proj ect 9 especial2 2`r because the firm can exploit its monopoly position to 8a, n even greater profits. However, the calculation is an ex post assessment. Ex ante, the priorate rate of return to such a proj act might have been very Low because of the risks associated witch private funding of sue-. a large proj ecu. From a social standpoint, the benefits of the rise reduction by ache federal government must be balanced with the possible side effects of the innovation. For example, there is the ~ oss of a rival firm that could have forced the surviving firm to maricet its products more competitively. Perhaps also, the rib a: --it To could have produced innovations that would have been useful socially. Similarly, there are potential benefits th.a~c can be realized b~ consumers. For instance, if the innovation is used later by commercial airlines, and it results in significant fuel savings that are passed on to the consumer in the form of tower fares, then perhaps those consumer benefi~cs should be included as part of the benefits of federal R&D allocated to j et engine research. The gene rat principle espoused above is that if one is crying to evaluate what the federal government received for its expenditures on R&D, then one must be careful to account not only for easily recognizable public and private expenditures and benefits' but also for spillover effec~cs and other ex~cernalit~es thee affect consumers and competitors. This is easier said than done. Nowhere are62the do ff~cul~ies more apparent than they are in Griliches ' study of the social rates of return to hybrid court and sorghum. Aside from being a fine example of the economic subtleties involved in accounting for the benefi~cs of federally suppor ted innovations . the paper also is introspective in cons idering the limits 0 f such calculations . Among the concep tual issues raised, the mos t re learnt for s~:udies of the efficacy of federal R&D are: (1) It is a difficult problem to draw the line between applicable and nonapplicable R&1). One must be sure to include both the successful and the unsuccessful R&D expended. Yet, if too much unsuccessful or "unrelated" R&D is included, one can come arbitrarily close to the estimate of a zero (or negative) rate of return. ~ 2) One must be careful not to make inferences that assume ;~.a. the average rates of return hold at the margin. — 98 —

~ 3 ~ Case Saudis provide useful information for specific i-.novat ons, but often the results are difficult to generalize. Gnat s because there is a ~ endency for such studies to exantin.e either exceedingly successful prod ects ~ for example, polio vacc' nes, ~ e aircraft, and hybrid corn) or dismal failures ~ for example, high-pressure nuclear reactor technologies ~ . SURVEY METHODS An alternative means for measuring ache impact of federal R&D expenditures is survey research This method of assess ing the contribution of federal programs to innovation and economic growth has ~ number of distinct advantages . I t asks directly for information about federal programs and their impacts; it allows economists to snake more qualitative assessments of federal programs (form example, in what ways federal programs promote output growth); and it provides information about how private alternatives compare o federally sponsored research Given the limited amount of survey work chat has been undertaken, it ~ s not surprising that such research has maj or limitations also . lathe most significant appear to be the cos ts invo rued in des igning and imp lamenting surreys 9 the difficulty in interpreting questions that provide qualm Satiate or subj echoic inforuaa~cion about returns . and the problem of response bias when many firms have a continuing contractual relet' onship Both federal agencies . Despite these limitations, a number of surveys Zaire asked questions dealing with the contribution of federal R&I) to such diverse areas as manpower programs, defense, sec~coral sp illovers, and the effec&~'reness of agency funding pro~ris ions ~ see S Sanford Research Institute and Clarke's bibliography for references) . Two recent surreys that may be of particular incest co federal pol~cymakers are those of Man field ~ and his associates and the Yale surrey (see Lenin et ale ). lathe Mansfield surreys asked ~ variety of questions relating to the effect of federal R&D on private energy R&D and the accuracy of federal engineering employment forecasters; he specific lessons learned from the surveys are s~mrnarized nicely in the Mansfield paper and are not repeated here. More generally, . the Mansfield surreys are important because they are con~rincislg in showing that there is important firm- leered detail thee the other methodologies overlook. Ig7par~cicular, his surveys and subsequent work by Foster Associates and Robert Nathan Associates68 have identified a number of characteristics associated with socially unproductive and productive inno~rattons. The Yale survey is interesting because of its scope. lathe authors surveyed corporations about the opportunity and appropriability conditions in a large number of markets. Their sample provides complete responses from 6SO business units representing more than 130 distinct three- and four-digit Standard Industrial Classification _ 99 _

~ SIC) Lines of business . Among the ques Lions asked were ones designed to evaluate the can~cr~butions of universe tv research, government research laboratories, and other government agencies to technological progress. Although the responses have eyed to be punt ° shed, it appears that there are fascinating differences both -within and across industries in the usefulness of the output of government research 4,bora~cories versus the output of other government agencies. Much fore work, however needs to be done to disentangle the information in the surrey results. MEASUREMENT A-5JO DATA ISSUES Each of the above techniques presumes tha~c federal R&D and what it does con be measured accurately. tot only do the techniques presume that indites and outputs can be measured accurate ly, but they also presume that the necessary data are readily a~raitabie. As federal R&D poses special measurement issues and data problems, this sects on summarizes areas in Which future s tudies 0 ~ federal R&3) are like 1~; to encounter difficulties. F~3 ERAL R&D INPUTS I ~ would appear easy to quantify the cost or amount of federal R&D that ~ s an input into an R&D process . However, there are many practical and conceptual issues that must be addressed before a sa.tisfactory input measure can be calculated. Although the precise fo mu the R&D input should take depends upon the particular app Lication, ehe fallowing general issues apply ~o most studies . Temporal Aggregation By and large, R&D is an investment activity that takes place over a period of years. Even if one can identify in what years the R&D occurred and had its effec~c as an input, it Is difficult to construct a s Cock of R&D measure . One of the largest problems in constructing the stock is that ~chere ace no very reliable R&D price deflators ~ see Griliches and l~ichtenberg ). Another problem is that it is difficult to identify points at which the R&D began and where it will end. For example, in trying to calculate the return to federal R&D spent on satellite tracking systems, one could consider going back as far as the development of radar. At ache other end, it must be realized that present research on an~c~satellite ~cechr~ologies benefits from precarious satellite tracking technology, and there may be more benefits to come. Finally, there is the question of whether stocks of R&D should be gross or net. As R&D depreciates economically or obsolesces with the introduction of new technologies and new — 100 —

~cuowiedge . the effect) He stock of R&D declines . Often, i- is ..o ~ ~ s imp le matter to es Climate depreciation races for knowledge, and .~_r ~- scudies simply suppress consideration of whether the races of they have calculated represent gross or net returns. Source Jesus Use De finis ions of Inputs Just as there is a quests on about the relevant time dimension to federal R&D, there also is an issue of how i nciusive an I&) measure should be . For example, to compute the social contribution o ~ federal 2&~:) to the design of high-speed photographic equi?rnen~ one rout d have co consider whether to include only ode R&D a13 oca~ed direct' y for that purpose, or whether to use a more inclusive definition that ~ oaks at all contributions to the de~relopment of photographic equipment. Thus, for example, special computer software designed to improve photographs taken by spy sanely it's could be counted as an input. Borrowed priorate R&D might also be included separately as a necessary input into the process of development. Bo th examp les are ins tances of sp rollovers where o the r R&D inputs ~ o r outputs) act either as a joint product or as a pecuniary external ~ ;. Such j oint products and externalities pose special dilemmas for studies of federal R&D. On one hand, they usually can be recognized only in micro- level studies of the effectiveness of federal R&D . Such studies then face the difficul ~ task of Ring to calibrate -he general significance of their impact. On the ocher hand, macro- level analyses tend to view such effects as residual spillo~rer. If they are acknowledged" at all, ~ ~ is usually through arguments that suggest aggregate measures of R&D proxy for the spil lover . Howe~rer, agricultural economists have acct,tnulated a large body of evidence that suggest~l~chat this is not necessarily a good assumption ~ see Evenson ~ . Other Defini~cionai,Problems It alas noticed earlier that the R&D input observed most often is a dollar amount and not a physical quantity. In the case of federal R6cO, there is a real issue of whether dollar amounts bear a consistent relationship to ache number and quality of the units of R&D purchased. For example, the R&D services bought with one dollar of contract R&D might well be very different from one dollar spent on a publicly managed proj ect. Not only are there such differences across federal proj ects by type of inputs purchased, but there is also a prob lem in knowing whether the government accounting definition of "R&D " matches what we would like to measure . For example, in the data reported by the various federal agencies and publicly managed proj ecus, there appears to be some question as to how to report such items as contract Rap) overhead, training programs, information72 pro~ris ion, experimentation, and data collection ~ see Gril~iches - 101 —

and lertecky; 73) . Then, coo, there is the problem of how (or whetters co value such public goods as the Library of. Congress. ~Y.E OL'T?~= OF rEDERAL R&D A myriad of conceptual and measurement problems is associated w' oh measuring the output of federal programs. Many problems are related to semantic questions such as: What is it that we would Like to measure? Or, what is it that we can measure? Such problems are compounded by the inherent unobservability of some of che effects of federal R&i), especially in the political and the social realms. Gina follows is an abbreviated list of measurement problems relevant to gauging the outputs of federal Rid). identification Issues Many of ache problems associated with measuring the output of federal R&D seem from the difficulty of identifying the consequences of such R&33. In part, there is the conceptual problem of defining outputs when it is unclear what physical quantity or service should be measured. There is also, however, ~ conceptual problem associated crick causall ~ relating federal expenditures deco the potential externalities they create. For example, it is not only difficult to quantify medical research conducted in ~leteranst Administration hospitals or the Walter Reed Armor Medical Center, but tic also is difficult deco predict whether the output from past research in those hospitals has yet to be realized. Currency, there are few models or procedures that could help decide these issues, and the problem can only get worse in such sectors as heal th, space, and defense . Not only is it difficult to quantify the direct output of R&D in sec~cors there there is no clear measure of goods and services, but there also is the problem that federal R&D often produces inputs into ouches ac~ci~rities. For instance, federal R&D allocated to medicine may produce teas tc knowledge that lancer has app locations in biotechnology. Another familiar example is federally supported research in abstract mathematics. Often, in ~cha~c discipline, there is no cisar index for the value of theorems, except perhaps that they may serve as inputs into other theorems that later may have some readily quantifiable impact. Lee underlying point is that when there is no immediate or recognizable nmaricet price" or "market use" for an output, the output may be undar~ratued or no t valued . Ano~cher problem economists have in identifying the output of R&D is that of uncovering and measuring sp:llo~rers and spinoffs. Federal research poses special problems in this regard because spillovers and sp inoffs occur over time and across markets . Federal research also tends to require public dissemination of its outputs. Thus, by its - 102 —

tern nature, federal,, R&D poses ~ probe era o' identifying indi'ec- , . Outputs ~ s ee Lowe ry ' ~ ~ rather teal Nation Issues As mentioned above, one of the maj or problems associated with identifying the output of federal R&D is that there may be no marine Is in which to obesin valuations of the output. Indeed, federal R&D often is allocated to certain areas precisely because there is a marks failure ~ for ins Dance, in areas where it is too CoSt:V for a Arced to exist or where a public good i s being provided and the pri rate valuation bears no relationship .o the social Valuation) . the pas t, the lack o f Parke e valuations o f Rap) output has led researchers co value outputs according to the inputs that produced them. For example, outputs in the health field come in physician hours or beds occupied He output of federal R&D labors tories is measured in terms of the numbers of outside contacts or numbers or s .udies conducted ~ see Charles River Associates " ~ . Such ~ cost~based definition of contributions is problematic, particular!-; in the defense and space sectors, because government contracts other. define in advance what the prices are. Overhead policies also can produce pathological valuations of RS3 output. Everyone has heard o r the specially designed $7, COO high^altitude coffee pars and $SOO s c rewdri~re rs . All of the problems described above do not inspire confidence our ability to measure the outputs of federal R&0 directly. The res idual ques tion then is, How bad are our measures for the uses '~e have in mind? This issue is talcen up in the conclus ion. SU=ARY ANI) PROSPECTUS This paper has summarized the techniques used to assess the returns to federal R&D. No doubt it is an incomplete surrey in Chat there are many ai Bed satire ways deco answer the ques tion, "'-~at are the returns to federal R&D? " What is left is an impression that we Cadre some answers but are still far short of reliable empirical statements. We also are left witch an impression that there are no easy shor~ccuts deco improve our methodologies dramatically. For example, it is clear that there are many R&D outputs that will never be quantifiable. There are also situations in which all. ache important consequences o f federal RScO may never be identified. The limitations that center on debates about what is and is not measured, and what techniques are and are not appropriate, often clash with the policy analyst's need for more refined statements about the contribu~cion of federal RScD to growth. It seems fair deco say that current economic measures of returns to federal Red) provide. — ~ (I 3 —

a~ most, crude historical s statements about the contributions or federal R&D. Thus, it ~ s dangerous to assume that the calcuiat~ ons can be used in predictive contexts. For example, current methodologies simply do not allow us to say that because the returns to energy R&D in the tate 1970's were miserable, we should expect them to be miserable in the future. Moreover, it is important to rest st comparisons of the historical statements that we do have. A prominent example of this is the tendency to compare es Reared rates of return to different government projects. Such comparisons are difficult because different methodologies measure different quantities. Often, incompatibilities of different methodologies ore overicoked in comparisons of rates of return. For example, economists rarely ask in such comparisons whether they are comcar'-.g average versus marginal rates of return, direct versus indirect effects, and pares ~1 versus genera)6egllilibri'~m analyses ~ cons ider, for examples the Fos racer As soc :~ Ices and the Robert Nathan Associates comparisons or races of return to different innovations ~ . At1 of these subtleties argue for more caution in using ex: stung cal~cula~cions . If there are problems in comparing race of retunes results across methodologies, then we need deco ask how the evidence in those stay es can best (comparatively) be interpreted. At a minims, it seems _h^- what present methodologies do is provide useful tools for anai~zi-5 the cnaracreriscics of proj ects that require federal support. They also, in principle, . cell something about historical rates of return to federal R&D, whether those returns Wearied by type of R&D (for example, basic versus applied), and whether they were higher or tower for similar applications. We must realize, however, that the weight assigned to the results should bear some relation to the quality of the data. That is perhaps discouraging, because the available date on federal R&D inputs, such as actual federal spending, are far from perfect. he do not know, for example, much about the Independent Research and Development (IR&~) allowance or- basic R&D. '~e also do not understand ouch about how the political process or government institutions affect the allocation and accounting of federal R&D. For example, a quick glance at the fine print of the Federal Procurement Regulations reveals that government negotiators have a fair amount of discsecton in determining what gets counted as contract overhead allowances. A lack of data on what those allowances may amount to mites it very difficult to interpret reported contract R&D funguses. On the R&D input side, better measures of the end uses of federal R&D are required. A s ignifican~c fraction of federal I&) is attached to readily identifiable prod acts or contracts, yet, typically, government pub lications organize expenditure data by issuing agency . There is some federal R&D contract data that is organized conveniently by appligge~on ~ for example, The 3 in ~ Research and Development Direcrory ), but it is unclear how complete those data are. On the output side, fairly detailed product information ~ s available for such sectors as agriculture and manufacturing. Lacking, however, are comparable data in such rapidly growing sectors — 104 —

as health and space, as well as detailed i-.fornation on he of go~rernmenc research laboratorie This List of data inadequacies could go on and on ( see also Gr~liches ~ . However, progress coward impro~rtr~g =he re fiance (and pLausibil' tin of economic studies of the contribution of Faders R&D requires snore than progress on data issues. Definitional problems and methodological problems deserve parallel a=tenelon. So doubt, many of the papers at this conference will help clarify what the research agenda should be. Much, however, remains to be done before ore can answer the detailed questions we ask of the data . - lob . ~

NOTES AND RE.- ERENCES Thus, the paper ~ Snores the consequences of other ways -:~ ant c-. federal policy affects technical change. For exempt e, ~~x ray es . regulation, and patent legislation are important ai~ernat be means bay which ache federal government can influence the rate and direction of technical change. Z'ri Gritiches. "Research E.xpendi~cures and Growth Accounting.'' In Sci ence and Technology in fE:cc~nomic Grow, . Edi ted by B . R. '~il~l^ams. London: Mac.Milian, 1973. Hi Grtliches . " Issues in Assess ing the Contribution 0 ~ Resee rc-. and Development Co Productivity Growth, 't Be 71 Journal of Economics, Voi . 10 ~ t979 ), pp . 92 -116 . Z`ri Griliches. "Returns to Research and Development Expenditures in the Private Sector." In New De~relop¢~encs in P-oducri~r=~v ."easurement and Analysis. Edited by J. Hendrick and B. ~7accara. Chicago: University of Chicago Press! L98C. 5 . E . Mansfield. ~ ndJuserial Research and Technological Innovation . New York: Norton, L~6 8 . E. .Mansf~eld en al. "Social and Pri~ra~ce Rates of Return From Industrial Innovations, " Quar' erly Journal of Economics, Sol. 91~3 977), pp ~ 211-240. N . E . Terlecky: . Effec as of R&D on the Produc t; vi tar Grow t.~2 of Indt2ser, es: An Expioracory Strider. Report Number 140 . Washington, OC: National Planning Association, 1974. . E . Terlecky: . Economic Effec as or Government ED Spendir.g in ache Unit ted Stares c Discuss ion Paper 8 - 31- 84 . Washington, OC: National. Planning Association, 1984a. 9. Griliches, 1979, op. cit. t0. Terleckyj, t984a, op. cit. 1~1. D. M. Levy and N. E. Terlec~rj. "effects of Government R&D on Priorate R6`D In~restmen~c and Productivity: A Macroeconomic Analysis, n Bell Journal of Economics, Sol. 14~1984) ,, pp. 5 51 - S61 . . This paper considers only those conceptual models that Cadre been used widely to estimate the contributions of corporate or federal Rue. For reviews of recent theoretical work on ache economics of R&D, see M. Kamien and N. Schwartz. Maricer Secure and Innovation. Cambridge, England: Cambridge Un~trersi~cy Press, 1982; and Martha Dasgupta. "lye Theory of Techno] ogical - 106 -

Competition." Forking Paper, London School of Economics. Unnumbered, 1-47. ~3. Few studies distinguish between the effects of federal R&3 and priorate R&D, and few recognize factors ocher than economic var~ab les . it. he deflation of R&D expenditures by an input price deflator leads to a measure of adj usted real R&D inputs . For addi Atonal observations on the intended and unintended consequences of that procedure, see ZO Griliches and F. Lichtenberg. Errors of `"easuremene in Our?ur Def labors . Mimeo, National Bureau or Economic Research Summer Ins-^tu~ce, 1984~. is. L. Goldberg. T he Influence of Federal RED Funding on The Demand For and Returns ro Induserial R&D. CHIC Report Number 3 8 8 . Alexand. ia, AJAR Center for Oral Analyses, Public Research Ins titute, L9 79 . t6. R. Nelson and S. Winter. An EvolurionarY Theory of Tec.~:ca: Change. Cambridge, ~MA: Harvard IJniversi~y Press, 1982. 17 . There is some formal and informal evidence to support such stories . Licheenberg (F. Lichtenberg a Row churn Pr'vareiv- .~ financed R&D is Jefense~Pelated? Himeo, Colombia Graduate School of Business, 198S) has some general evidence for defense contracts. A recent Department of Justice suit charges ~ division of GTE Corps with gain" so far as to take defense department planning memoranda to discover what future proj acts are p tanned. :8. Economists lean toward quantification because i ~ provides benchmarks by which they can compare the contributions of different programs, funding provisions, agencies, etc . Often, economists are attacked for being too ready to quantify that which should not be quantified. That issue is far beyond the scope of this paper. However, a related question discussed here is whether such calculations can be compared meaningfully across studies . . Nestor E. Tertec~j. "Measuring Economic Effects of Federal Research and Development Expenditures: Resent History with Special Emphasis on Federal R&D Performed in Industry," this conference . 20. See, for example, the surrey by Griliches, 1979, op. click 21. For recent evidence on these issues and a discuss ion of related problems, see M. Tra; tenberg. Measuring cbe Welfare Gains From Product Innovations. Mimea, National Bureau of Economic Research Summer Institute, 1985. — iO7 ~

29. Tern eckyj, 1984a, op. cic. 23. One exception is Levy and Terteckyj, op cic. See also Terlecicyj, 19 84a, op . c i ~ . 24. Zip Griliches. "lithe Sources of Measured Produc~i~:icy Growth, U. S . Agriculture t940- i960, " Journal or Political Economy, (1963), pp. 331-346. 25 . R. E,renson. The Cont~:bu~ion of Agricul rural Researc.h and Extension co Agricultural Production. Ph. D. Thesis, l~ni~rersity of Chicago, 1968. 25 . J . Minasian. "Research and Development, Production Func ~ ons and Rates of Return," American Economic Review, Vot. 59~1969), pp. 80-85. Teriecicyj, 1974, up. ci-., pp. 1~12. 28. Gril~ches, t979, op. cite. 29. GriLiches, ~ 963, op. cat. 3 0 . Evenson, op . c i- . 31. Mansfield, 1968, op. cit. 3 2 O Terleckyj, 19 74, op . c i ~ . 3 3 . Levy and lerlecicyj, op . c i t.. 3~. Z. Griliches and F. Lichrenberg. "~&D and Productivity Growth at the Firm Levels Is there Still a Relationship?" In R&D, Percents, and Product~vi Cy. Edited by Zvi Griliches . Chicago: Uni~rersi~cy of Chicago Press, 1984b. For example, labor inputs, as measured conventionally, often include scientific personnel. 3 6 . E . Mansfield. "R and ~ Inno~rat~on: Some Empirical Findings . " In R&D, Parents, and Productivity. Edited by Zvi Griliches . Chicago: Unifiers ity 0 f Chicago Press, L9 84 . 37. Ma Schanke~man. "~he Effects of Double-Coun~cing and Expensing in the treasured Returns to RED, n Review Of Economics and Statistics, Vol. 63 (1981), pp . 454-458 . 38 . R. Levin. "=e Semiconductor Industry. " In Government and Technical Progress. Edited by R. Nelson. New York: Pergamon Press, 1982. ~ 108 -

39. A. S. Link. "An Analysis Of thee Composition of R&D Scenc.ng ' Southern Economic Journal, Jo 1. 48 ( 19 82 ), pp . 342 - 349 40. R. Levin and P. Reiss. "Tests of a Sch~peCerian Model of Market S tructure . '' In R&D f Parents, and Produc Nisi cv. Edited by Sari Griliches. Chicago: Un' Persia of Chicago Press. L984 . pp . t75 - 204. J. Scotch. 'Grimm Versus Industry Variability in R&D Intensity ' In R&D, Parenes, -and Product:tr:~. Edited by Z~ri Griliches. Chicago: Un~rer~ity of Chicago Press, i984. 42. Lew and Terlecicy:, op. cic. 43. Mansfield, 1984, op. cit. 44 . D ~ flowery . ''Economic Theory and Ga~rernment Techno logical Potic-<," Policy Sciences, Vot. i6('983), pp. 27~43. 4: . D c flowery and N. Rosenberg ~ "The Commerc ial Aircraft Industry. " In Governmene =2d Tec.Lmica1 Progress. Edited by R. Nelson. Sew York: Pergamon Press, t982. 46 . Lichtenberg, op . cit. ~ See Reference 17 above) 47. flowery, op. cite. 48. Mansfield, 1984, op. cit. 49. Link, op. cite. SO. Scott, op. cit. SLo Levin and Reiss, op. cit. 52. Terleckyj, 1984a, op. cit. 5 3 . Levy and lerleckyj, op . cit . :4 . Charles River Assoc fates, Inc . Produc rivi cy Impac Is or Government R&D Laboratories: The .~Jet~ona1 Bureau of Standards Semiconduc~cor Technology Program. Report submi~cted to the Depar~cmen~c of Commerce, National Bureau of S tandards, 1981 . So. Gril'ches and L~ch~cenberg, i984b, op. chic. ; 6 . C . Horugren. Cc~st Account ing: A Managerial Emphasis . Englewood Cliffs, Jo: Prentica-Ha] 1, t982. 57. S. L. Ansani. "Accounting and Reporting R&D Costs--Current Prac rices and Prob 1 ems, " Research Managemen A, To 1 . 19~1976), pp. 28-34. _ ° 109 °

:3 . T' bid. :9. A. H. Rubenstein. "Economic Evaluate on of Research and Development: A Brief Review or Theory and Practice." Jou_.-a o" Industrial Engineering, Vol. LO 366), pp. 61~-620. S. E. Ierleckyj . 'tA Growth .~£ode: of the U. S. Communication Industry." In ~n~or;nacion and Communications Economies: .Ve~v Perspectives. Edited bar M. Jussawalla and H. Ebenfiel~d. Amsterdam: borsch Holland, 1984b. Terleckyj, 1084~. op. ci.. 52. Zero Griliches. "hybrid Corn: Art Exploration in ~ he Economy cs of Technological Change . 't c conorzerr~ca, Vo1. 2S ~19~8 ~ . UP . 4i9 -43 1. 63 . The Economic Impace of Defense R&0 EoYpenCt, Cures. Menlo Paric, CA: Stanford Research Ins~ci. ute, L966 . 64 ~ T. E. Clarice . R&D .`lanagemen~ ~ ibl.ograchv. Third edition. lancouver , BC: S targa-e Consultants Led., L98' . Mansfield, 1984, op. cat o6 . R. r Turin, A. Klevorick9 R. Nelson, and S. Winter. "Survey Research on Appropr~ability Technological Opportunity: Part I n t' Mimeo, Yale Uniters icy Economics Department, 1984, pp . t-31. 6 7 . Foster Associates, Inc . "A Surrey of yet Rates of Return on Innovations. " Report submitted to the National Science Foundation, t978. 68. Robert R. Nathan Associates, Inc. "Net Races of Return on Innovations. " Report submitted to the National Science Foundation, 1978. 6 3 . These s tatements are made on the teas is o r bra liminary anal3rs es by Richard Lenin and Peter Reiss. 70 . Gril~iches and Lichtenberg, 1984a, op . chic . ~ See Reference 14 above ~ R. Evenson. '!A Century of Producti~rity Change In U.S. Agriculture: An Analysis of the Role of Invention, Research and Extension. " Yale Economic Growth Center Discussion Paper Number 296, 1Q78. 72. Griliches, 15379, op. cit. Terleckyj, 1984a, op. cf. — 1.0 —

- ~ 75. 76. 77 gone ry, op Charles River Associaces, Foster Associaces, Inc., op Inc., op. Roberc R. Nachan Associaces, Inc., op. cic. 78. For issues refacing co IR6D, 79. see Terleckyj , 1984a, op. See, for example, Federal Procu~ecenC ~equlacions 1-15.204-35 and 1~15.1Q7. , sections SO. :he ~ in ~ Research and Deveiopcanc Directory. Washington, DC: Gover~menc Caca Publicacions, 81. Criliches, 1984. 1979, op cic. 111 -