National Academies Press: OpenBook

Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program (2005)

Chapter: Appendix C Pool of Generic Metrics for Science and Technology

« Previous: Appendix B Case Study Metrics for the Climate Change Science Program
Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×

Appendix C
Pool of Generic Metrics for Science and Technology

A generic set of metrics for evaluating science and technology programs has been developed by E. Geisler, based on a review of the literature.1 The following metrics are the subset that is most relevant to an agency research and development (R&D) program, and they are categorized according to the committee’s definitions of process, input, and output metrics. Geisler did not identify generic outcome or impact metrics.

PROCESS METRICS

Organizational, Strategic, and Managerial Metrics
  1. Internal or cycle time: period from the start of a project to transferring an outcome to a downstream unit within the organization.

  2. External or commercial cycle time: period from the start of the project to the ultimate sale of a product or service to an external customer.

  3. Existence of project champion: number or portion of current projects that have an identifiable champion in the form of a manager from outside the R&D unit.

1  

Geisler, E., 1999, The metrics of technology evolution: Where we stand and where we should go from here, Annual Technology Transfer Society Meeting, July 15-17, 1999, <http://www.stuart.iit.edu/faculty/workingpapers/technology/>.

Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
  1. Projects with interfunctional teams: number of projects that employ teams composed of people from units across the organization and outside the R&D unit.

  2. Evaluation of the scientific and technical capabilities of the R&D unit and, by extension, of the total organization: external evaluation primarily by various customers, of the capability of the firm and its R&D unit in meeting the scientific and technological challenges of changing markets.

  3. Project progress and success: progress in meeting established objectives and milestones over a given period of time; number or percentage of projects that exhibited technical success on time and on budget.

  4. Evaluation of projects and programs: averages of cost per project, by type of project.

  5. Ownership, support, and funding of projects and programs: percentage of projects supported and funded by other units in the organization that are directly related to a product line or similar commercial entity in the organization; distribution of projects and programs by source of organization.

  6. Human relations measures of R&D personnel: morale of personnel; satisfaction with their work.

  7. Relation of R&D to strategic objectives: degree to which R&D objectives are related to the strategic objectives of the organization and are current with any changes in the organization’s strategy.

  8. Benchmarking project and program performance: relation of project management metrics to benchmarks that are standards, averages, or best practices in the industry or sector; extent to which these benchmarks influence the strategic direction of both R&D and the total organization.

Peer Review Metrics
  1. Internal evaluation: subjective rating by other people in the organization ranked on a scale that measures judgment of respondents.

  2. External evaluation: subjective evaluation by a panel of experts.

  3. Targeted reviews: panel evaluations of any R&D outcome. This may be considered a measure of quality, as viewed by expert reviewers.

INPUT METRICS

Investments in R&D
  1. Expenditures for each stage of research and development.

  2. Expenditures per time frame, for one time period, or over several time periods.

  3. Distribution by categories of expenditures for personnel, equipment, et cetera.

Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
  1. Source of funding.

  2. Comparison of expenditures, per item category, by competitors, industry averages, and sector averages.

  3. Expenditures by discipline, technology, and scientist and engineer.

  4. Expenditures related to a product line or other commercial unit of reference, such as customer or market.

OUTPUT METRICS

Bibliometric Measures
  1. Publications.

  2. Citation analysis.

  3. Co-word analysis and database tomography: analyses performed on large databases of R&D bibliographical outcomes, in a form of data mining.

  4. Special presentations and honors.

Stages of Outcomes
  1. Immediate outputs: proximal or direct outputs from the R&D activity, such as bibliometric measures.

  2. Intermediate outputs: outputs of the organizations and entities that have received the immediate outputs, transformed them, and are providing the transformed outputs to other entities in society and the economy.

  3. Pre-ultimate outputs: products and services that are generated by those social and economic entities that have received and transformed the intermediate outputs.

  4. Ultimate outputs: things of value to the economy and society that were impacted by the pre-ultimate outputs.

  5. Index of leading indicators: weighted measures of core and organization-specific measures intended to provide a quantitative appraisal of the value of R&D at each stage of the innovation process.

  6. Value indices for leading indicators: value of each index at each stage of the innovation continuum. Value indices are computed by subtracting the value of each leading index from the index that succeeded it. Net value is computed by comparison with costs of R&D and transformation at each stage.

  7. Portion of R&D at each stage: the role that R&D has in each of the stages for each of the recipient or transforming organizations. These measures offer a look at the size and value of the R&D contribution for each output, as well as in toto.

Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×

This page intentionally left blank.

Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
Page 141
Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
Page 142
Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
Page 143
Suggested Citation:"Appendix C Pool of Generic Metrics for Science and Technology." National Research Council. 2005. Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program. Washington, DC: The National Academies Press. doi: 10.17226/11292.
×
Page 144
Next: Appendix D Biographical Sketches of Committee Members »
Thinking Strategically: The Appropriate Use of Metrics for the Climate Change Science Program Get This Book
×
Buy Paperback | $52.00 Buy Ebook | $41.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

The Climate Change Science Program (CCSP) and its predecessor U.S. Global Change Research Program have sponsored climate research and observations for nearly 15 years, yet the overall progress of the program has not been measured systematically. Metrics—a system of measurement that includes the item being measured, the unit of measurement, and the value of the unit—offer a tool for measuring such progress; improving program performance; and demonstrating program successes to Congress, the Office of Management and Budget, and the public. This report lays out a framework for creating and implementing metrics for the CCSP. A general set of metrics provides a starting point for identifying the most important measures, and the principles provide guidance for refining the metrics and avoiding unintended consequences.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!