Appendix C
Pool of Generic Metrics for Science and Technology
A generic set of metrics for evaluating science and technology programs has been developed by E. Geisler, based on a review of the literature.1 The following metrics are the subset that is most relevant to an agency research and development (R&D) program, and they are categorized according to the committee’s definitions of process, input, and output metrics. Geisler did not identify generic outcome or impact metrics.
PROCESS METRICS
Organizational, Strategic, and Managerial Metrics
-
Internal or cycle time: period from the start of a project to transferring an outcome to a downstream unit within the organization.
-
External or commercial cycle time: period from the start of the project to the ultimate sale of a product or service to an external customer.
-
Existence of project champion: number or portion of current projects that have an identifiable champion in the form of a manager from outside the R&D unit.
1 |
Geisler, E., 1999, The metrics of technology evolution: Where we stand and where we should go from here, Annual Technology Transfer Society Meeting, July 15-17, 1999, <http://www.stuart.iit.edu/faculty/workingpapers/technology/>. |
-
Projects with interfunctional teams: number of projects that employ teams composed of people from units across the organization and outside the R&D unit.
-
Evaluation of the scientific and technical capabilities of the R&D unit and, by extension, of the total organization: external evaluation primarily by various customers, of the capability of the firm and its R&D unit in meeting the scientific and technological challenges of changing markets.
-
Project progress and success: progress in meeting established objectives and milestones over a given period of time; number or percentage of projects that exhibited technical success on time and on budget.
-
Evaluation of projects and programs: averages of cost per project, by type of project.
-
Ownership, support, and funding of projects and programs: percentage of projects supported and funded by other units in the organization that are directly related to a product line or similar commercial entity in the organization; distribution of projects and programs by source of organization.
-
Human relations measures of R&D personnel: morale of personnel; satisfaction with their work.
-
Relation of R&D to strategic objectives: degree to which R&D objectives are related to the strategic objectives of the organization and are current with any changes in the organization’s strategy.
-
Benchmarking project and program performance: relation of project management metrics to benchmarks that are standards, averages, or best practices in the industry or sector; extent to which these benchmarks influence the strategic direction of both R&D and the total organization.
Peer Review Metrics
-
Internal evaluation: subjective rating by other people in the organization ranked on a scale that measures judgment of respondents.
-
External evaluation: subjective evaluation by a panel of experts.
-
Targeted reviews: panel evaluations of any R&D outcome. This may be considered a measure of quality, as viewed by expert reviewers.
INPUT METRICS
Investments in R&D
-
Expenditures for each stage of research and development.
-
Expenditures per time frame, for one time period, or over several time periods.
-
Distribution by categories of expenditures for personnel, equipment, et cetera.
-
Source of funding.
-
Comparison of expenditures, per item category, by competitors, industry averages, and sector averages.
-
Expenditures by discipline, technology, and scientist and engineer.
-
Expenditures related to a product line or other commercial unit of reference, such as customer or market.
OUTPUT METRICS
Bibliometric Measures
-
Publications.
-
Citation analysis.
-
Co-word analysis and database tomography: analyses performed on large databases of R&D bibliographical outcomes, in a form of data mining.
-
Special presentations and honors.
Stages of Outcomes
-
Immediate outputs: proximal or direct outputs from the R&D activity, such as bibliometric measures.
-
Intermediate outputs: outputs of the organizations and entities that have received the immediate outputs, transformed them, and are providing the transformed outputs to other entities in society and the economy.
-
Pre-ultimate outputs: products and services that are generated by those social and economic entities that have received and transformed the intermediate outputs.
-
Ultimate outputs: things of value to the economy and society that were impacted by the pre-ultimate outputs.
-
Index of leading indicators: weighted measures of core and organization-specific measures intended to provide a quantitative appraisal of the value of R&D at each stage of the innovation process.
-
Value indices for leading indicators: value of each index at each stage of the innovation continuum. Value indices are computed by subtracting the value of each leading index from the index that succeeded it. Net value is computed by comparison with costs of R&D and transformation at each stage.
-
Portion of R&D at each stage: the role that R&D has in each of the stages for each of the recipient or transforming organizations. These measures offer a look at the size and value of the R&D contribution for each output, as well as in toto.