The National Science Foundation's Materials Research Science and Engineering Centers Program Looking Back, Moving Forward (2007) / Chapter Skim
Currently Skimming:
3 Assessment of Research and Facilities Impact
3 Assessment of Research and Facilities Impact
Pages 71-117
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 71... ...
research is presented in this chapter. INTRODUCTION The Materials Research Science and Engineering Centers program (MRSEC program)
|
From page 72... ...
and well-trained people." Evaluating MRSEC research is a daunting task. The committee considered sev eral strategies, realizing that the MRSEC program contributes to the NSF mission in multiple ways even though "short-term research results" are usually considered the primary objective (see Box 3.1)
|
From page 73... ...
FIGURE 3.1 Distribution of annual Materials Research Science and Engineering Center budgets in the 1990s (top) and in the current decade (bottom)
|
From page 74... ...
For instance, it would be insufficient to observe, "Research conducted through the MRSEC program gener ally includes significant collaboration." Rather, the committee sought to determine
|
From page 75... ...
4Available online at http://cresmet.asu.edu/nagps/about/index.php. if the rate or nature of collaboration in the MRSEC program is different from the rate or nature of collaboration outside the program.
|
From page 76... ...
Although the committee was unable to identify MRSEC-enabled research in "blind taste tests," it successfully assessed the overall research quality in comparison with the research enabled by other mechanisms and elsewhere around the world. The basic ques tions to be answered are whether the research enabled by the MRSEC program is distinctive, if it is worthwhile and of high quality, and, finally, whether it is a good investment.
|
From page 77... ...
While it was difficult to separate research uniquely enabled by the MRSEC program from research that was made possible by other means, the committee was clearer about causation. For instance, many of the more recently established NSF Nanoscale Science and Engineering Centers (NSECs)
|
From page 78... ...
The committee then identified where the research leading to each discovery had been done and, in particular, whether it had originated in the MRSEC program or its predecessor MRL program. The list contained very few items that occurred in the past decade and thus the discoveries on the list significantly predated the MRSEC program per se.
|
From page 79... ...
5 4 3 2 1 Number of Discoveries 0 s e y ls s ls ls er cs cs es g i ity i c er bl es a ia d th iv ria ni ra st rti on on xi er O e t o r ym ct em l ro ry o O t at u s pe at nt St i C d o Po ec Ph As d om om gy Pr on Sp El lf ui Bi rc er rt an c q e o Se N ni Li En a sp up S rg an O Tr Subfield FIGURE 3.2 Distribution of selected major materials research discoveries, by materials research subfield. A handful of the 27 discoveries identified by the committee did not fit into these categories and are labeled "Other." 3-2 9
|
From page 80... ...
No statistical analysis of these fragmentary observations is possible; however, it is pos sible to say that there are discoveries of the highest significance occurring within the MRSEC program, as gauged by this subjective survey. As can be seen later in this section, there is almost an orthogonality between the types of institutions responsible for "major discoveries" and "top cited papers," the former originating in industrial laboratories and the latter in universities.
|
From page 81... ...
14 12 4+ 20% 1 10 32% 3 8 8% 6 Number of Discoveries 4 2 2 40% 0 U.S. Universities MRSEC U.S.
|
From page 82... ...
. This highlight of the MRL/MRSEC program is on the list of selected major materials discoveries (see Box 3.5 for other examples of MRSEC research)
|
From page 83... ...
This collaborative research was conducted at an MRL and involved strong interactions between the physics and chemistry departments at Pennsylvania State University and a group in Japan; it led to new materials, conducting polymers, and new con
|
From page 84... ...
-- Of the top 20 most highly cited articles, 5 came from institutions with MRSECs. • Stripes in high-temperature superconductivity: J.M.
|
From page 85... ...
That is, truly eminent articles generally take 10 to 15 years to demonstrate their impact on the field.3 Thus, the committee's efforts to assess the research impact of the MRSEC program through a study of its publication citations is a bit premature. In its defense, the committee chose to compare the MRSEC program to just the past decade of materials research papers in order to include the same systematic error in the reference case.
|
From page 86... ...
Materi als research papers arising from institutions with MRSECs do not necessarily come from MRSEC enabled research. However, the committee could not find a better alternative.
|
From page 87... ...
. For comparison, the number of highly cited papers by subfield, for the set of "top 5" papers reported by the Materials Research Science and Engineering Centers, is included as the second bar (dark)
|
From page 88... ...
3.5 noteworthy that institutions with an MRL/MRSEC program accounted for about 10 percent of the most highly cited papers worldwide and about 20 percent of those from the United States. These are considerably larger percentages than might be indicated by the relative funding levels mentioned earlier.
|
From page 89... ...
The comparison indicates a strong correspondence between high-impact research done in the MRSEC program and the interests of the materials community as a whole. The committee also independently examined the current MRSEC research portfolio.
|
From page 90... ...
The average of this histogram is about 350. The top 100 most-cited papers had an average of 892 citations per paper, with a median of 775.
|
From page 91... ...
Bernhard Keimer at the Max Planck Institute in Stuttgart kindly offered the services of his library staff to compare citations for several Max Planck Institutes (MPIs) and the MRSEC program.
|
From page 92... ...
. Comparison with Figure 3.4 indicates that there is substan tial overlap of the most active areas with the corresponding efforts in the MRSEC program.
|
From page 93... ...
16,000 14,000 12,000 10,000 8,000 6,000 4,000 Number of Citations 2,000 0 .
|
From page 94... ...
of highly cited papers, followed by Europe (28 percent) and Asia (12 percent)
|
From page 95... ...
Other, 10, 1% Industry, 102, 3-9 12% Government, 159, 18% Academia, 598, 69% FIGURE 3.10 Performers of top-cited materials research summed over the 23 subfields, 1995-2006: percentage by type of institution.
|
From page 96... ...
It is difficult to assign agency ownership to a discovery, a new material, or even just a publication. MRSEC, 35, 4% Other, 153, 15% NSF, 216, 22% DOE, 96, 10% Non-U.S., 222, 22% NIH, 33, 3% DOD, 170, 17% Industry, 73, 7% FIGURE 3.11 Sources of funding for top-cited papers from U.S.
|
From page 97... ...
Overall, these data indicate that 65 percent of all top-cited papers involve multiple senior authors; however, Figure 3.13 shows that the most likely collaboration is between pairs of senior authors, with vanishing incidence of collaborations with three or more senior authors. There is no evidence that topcited papers by MRSEC investigators display a different trend, although a primary argument used to rationalize the MRSEC organization is that these larger-scale collaborations are only possible in centers that include a large number of researchers from different departments.
|
From page 98... ...
P e M pr an qu er an ol Se ot er N ne nu ec N Li at M En Tr Su up ra Ph ag M M g/ t. S r G a M O M Subfield FIGURE 3.12 Comparison of the number of top-cited papers by single or multiple senior principal investigators (PIs)
|
From page 99... ...
DEMOGRAPHICS OF RESEARCH PERFORMERS A primary objective of the original MRL program that has continued into the MRSEC program is to provide a setting that stimulates and nurtures interdisciplinary collaborative materials research. The original MRLs were located at institutions that already had a substantial interdisciplinary materials effort.
|
From page 100... ...
A generation after the establishment of the MRLs, there is no question that MRSEC research remains both broad and multidisciplinary, and perhaps one can make the argument that this intrinsic attribute of the MRL and MRSEC programs has led the trend in materials research more generally. The committee finds, therefore, that on the metric of multidisciplinarity as measured by departmental affiliation on research papers, the MRSEC program performs similar to the overall materials research community.
|
From page 101... ...
100 90 80 70 60 Physics Math 50 Biology Percentage Chemistry 40 Other Engineering Electrical Engineering 30 Materials 20 10 0 2004 1995 1975 Year FIGURE 3.15 Individual academic departments by discipline as a percentage of total departments participating in the Materials Research Laboratory and Materials Research Science and Engineering Centers programs. The most recent information, from 2004, is on the left-hand side of the figure.
|
From page 102... ...
It is unlike the previous citation results, which were broken down into MRSEC subareas. As shown in Figure 3.17, the percentage of top-cited papers is plotted for U.S.
|
From page 103... ...
The order from top to bottom on the chart is shown in the legend. NOTE: MRL, Materials Research Laboratory; MRSEC, Materials Research Science and Engineering Center.
|
From page 104... ...
THE LEADING GROUPS IN MATERIALS RESEARCH To assess the perceived excellence of the programs in 23 different subfields of materials research that may show differential MRSEC impact, the committee undertook an informal survey of the opinions of experts. It was proposed initially that the experts be selected by choosing the senior authors of the top 10 most-cited papers from each of the 23 subfields; however, the committee decided that the list should be augmented by authors of highly cited papers who were not selected by the simple algorithm above.
|
From page 105... ...
The committee notes that the membership of the Materials Research Society is not broadly reflective of the overall composition of the materials research community, but it does have certain parallels to the university-based research community of Materials Research Science and Engineering Centers. (Courtesy of the Materials Research Society.)
|
From page 106... ...
t h e n at i o na l s c i e n c e f o u n dat i o n ' s m r s e c P ro g r a m 06 United States Foreign Sources 33% 36% MRSEC 31% FIGURE 3.19 Sources of "expert votes" in the survey of leading research groups in materials. The "United States" category does not include the votes from institutions with a Materials Research Sci ence and Engineering Center (labeled "MRSEC" in the chart)
|
From page 107... ...
Furthermore, the exercises were sensitive to different characteristics of research excellence: major discoveries versus overall high quality. The distribution of leading groups was not uniform across the MRSEC program.
|
From page 108... ...
Alto gether, MRSEC expenditures represent a very small fraction of the federal materials portfolio, amounting to about 2% of the total. Of course, given the variety of activities funded by this portfolio and the dif ferent programmatic needs of the different agencies, there is no a priori reason to believe that the number of top-cited papers claimed by a given funding agency is proportional to its relative level of materials funding.
|
From page 109... ...
The MRSEC program (2% of total materials investment, 5% of top-cited papers) is similar in "efficiency" to NSF overall (14.6% of total materials investment, 30% of top-cited papers)
|
From page 110... ...
Within these statistics, there is little evidence that the MRSECs are more or less productive in this respect than any other NSF materials program. SHARED EXPERIMENTAL FACILITIES An often-cited key element of the MRSEC program is its explicit provision of shared experimental facilities (SEFs)
|
From page 111... ...
FIGURE 3.24 Correlation plot of total Materials Research Science and Engineering Center annual budget versus Shared Experimental Facilities budget. As expected, the correlation is positive but not linear.
|
From page 112... ...
In terms of impact, the committee believes that the shared facilities supported by MRSECs do have significant impact in the larger community, but the committee was not convinced that the MRSEC SEF support was dramatically more effective or leveraged more than any other instrumentation program. For instance, the com mittee learned from the Midsize Facilities report referred to above that operating costs for shared facilities (including the MRSEC program)
|
From page 113... ...
If MRSECs did not do this, DMR would need to create some other strong facilities program to support materials research. The MRSEC facilities budget also supports (at least in part)
|
From page 114... ...
FINDINGS AND RECOMMENDATIONS Conclusion: Consistent with previous analyses, the committee found no simple, quantitative, objective measure to clearly differentiate the MRSEC research product from that of other mechanisms supporting materials sci ence and engineering research. The committee found the task of evaluating the impact of MRSEC research quite daunting, primarily because research papers published in peer-reviewed 9National Research Council, Midsize Facilities: The Infrastructure for Materials Research, Washing ton, D.C.: The National Academies Press, 2006, p.
|
From page 115... ...
For instance, do published research results that acknowledge MRSEC resources achieve citation indices and other measures of impact comparable with research enabled by individual-investigator awards? Conclusion: Overall, the MRSEC program produces excellent, frontier sci ence of the same high standard as that supported by NSF through other mechanisms.
|
From page 116... ...
To some extent this may be the ultimate success of the MRSEC program in having fostered this type of research at an early stage. Finally, the breakdown of departmental affiliations of MRSEC authors and those of the top-cited materials research papers were quite similar.
|
From page 117... ...
As described in the beginning of this chapter, the committee concludes that the merit of the research enabled by the MRSEC program is comparable with the best of the materials research supported by other mechanisms. The committee notes, however, that it focused on measuring the impact of research results and that the ancillary benefits of MRSECs are not reflected by these metrics.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.