Improving Measurement of
Productivity in Higher Education
Panel on Measuring Higher Education Productivity:
Conceptual Framework and Data Needs
Teresa A. Sullivan, Christopher Mackie, William F. Massy, and
Esha Sinha, Editors
Committee on National Statistics
Board on Testing and Assessment
Division of Behavioral and Social Sciences and Education
NATIONAL RESEARCH COUNCIL
OF THE NATIONAL ACADEMIES
THE NATIONAL ACADEMIES PRESS
Washington, D.C.
THE NATIONAL ACADEMIES PRESS • 500 Fifth Street, NW • Washington, DC 20001
NOTICE: The project that is the subject of this report was approved by the Governing Board of the National Research Council, whose members are drawn from the councils of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the committee responsible for the report were chosen for their special competences and with regard for appropriate balance.
This study was supported by grant number 5793 between the National Academy of Sciences and Lumina Foundation. Support for the work of the Committee on National Statistics is provided by a consortium of federal agencies through a grant from the National Science Foundation (award number SES-1024012). Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the view of the organizations or agencies that provided support for this project.
International Standard Book Number-13: 978-0-309-25774-9
International Standard Book Number-10: 0-309-25774-3
Library of Congress Cataloging-in-Publication data are available from the Library of Congress
Additional copies of this report are available from the National Academies Press, 500 Fifth Street, NW, Keck 360, Washington, DC 20001; (800) 624-6242 or (202) 334-3313, http://www.nap.edu.
Copyright 2012 by the National Academy of Sciences. All rights reserved.
Printed in the United States of America
Suggested citation: National Research Council. (2012). Improving Measurement of Productivity in Higher Education. Panel on Measuring Higher Education Productivity: Conceptual Framework and Data Needs. Teresa A. Sullivan, Christopher Mackie, William F. Massy, and Esha Sinha, Editors. Committee on National Statistics and Board on Testing and Assessment, Division of Behavioral and Social Sciences and Education. Washington, DC: The National Academies Press.
THE NATIONAL ACADEMIES
Advisers to the Nation on Science, Engineering, and Medicine
The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished scholars engaged in scientific and engineering research, dedicated to the furtherance of science and technology and to their use for the general welfare. Upon the authority of the charter granted to it by the Congress in 1863, the Academy has a mandate that requires it to advise the federal government on scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences.
The National Academy of Engineering was established in 1964, under the charter of the National Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its administration and in the selection of its members, sharing with the National Academy of Sciences the responsibility for advising the federal government. The National Academy of Engineering also sponsors engineering programs aimed at meeting national needs, encourages education and research, and recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National Academy of Engineering.
The Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the services of eminent members of appropriate professions in the examination of policy matters pertaining to the health of the public. The Institute acts under the responsibility given to the National Academy of Sciences by its congressional charter to be an adviser to the federal government and, upon its own initiative, to identify issues of medical care, research, and education. Dr. Harvey V. Fineberg is president of the Institute of Medicine.
The National Research Council was organized by the National Academy of Sciences in 1916 to associate the broad community of science and technology with the Academy’s purposes of furthering knowledge and advising the federal government. Functioning in accordance with general policies determined by the Academy, the Council has become the principal operating agency of both the National Academy of Sciences and the National Academy of Engineering in providing services to the government, the public, and the scientific and engineering communities. The Council is administered jointly by both Academies and the Institute of Medicine. Dr. Ralph J. Cicerone and Dr. Charles M. Vest are chair and vice chair, respectively, of the National Research Council.
PANEL ON MEASURING HIGHER EDUCATION PRODUCTIVITY: CONCEPTUAL FRAMEWORK AND DATA NEEDS
TERESA A. SULLIVAN (Chair), Office of the President, University of Virginia
THOMAS R. BAILEY, Institute on Education and the Economy and Community College Research Center, Teachers College, Columbia University
BARRY P. BOSWORTH, Economic Studies Program, The Brookings Institution, Washington, DC
DAVID W. BRENEMAN, Curry School of Education, University of Virginia
RONALD G. EHRENBERG, Cornell Higher Education Research Institute, Cornell University
PETER T. EWELL, National Center for Higher Education Management Systems, Boulder, CO
IRWIN FELLER, Department of Economics (emeritus), Pennsylvania State University
BARBARA FRAUMENI, Muskie School of Public Service, University of Southern Maine
JULIET V. GARCIA, Office of the President, University of Texas at Brownsville and Texas Southmost College
MICHAEL HOUT, Department of Sociology, University of California, Berkeley
NATE JOHNSON, HCM Strategists, Washington, DC
GEORGE D. KUH, Center for Postsecondary Research (emeritus), Indiana University
WILLIAM F. MASSY, Independent Consultant, Florence, MA
CAROL A. TWIGG, National Center for Academic Transformation, Saratoga Springs, NY
DAVID J. ZIMMERMAN, Department of Economics, Williams College
CHRISTOPHER D. MACKIE, Study Director
STUART ELLIOTT, Senior Program Officer
ESHA SINHA, Associate Program Officer
MICHAEL SIRI, Program Associate
COMMITTEE ON NATIONAL STATISTICS 2011-2012
LAWRENCE BROWN (Chair), Department of Statistics, The Wharton School, University of Pennsylvania
JOHN M. ABOWD, School of Industrial and Labor Relations, Cornell University
ALICIA CARRIQUIRY, Department of Statistics, Iowa State University
WILLIAM DuMOUCHEL, Oracle Health Sciences, Waltham, MA
V. JOSEPH HOTZ, Department of Economics, Duke University
MICHAEL HOUT, Survey Research Center, University of California, Berkeley
KAREN KAFADAR, Department of Statistics, Indiana University
SALLIE KELLER, University of Waterloo, Ontario, Canada
LISA LYNCH, Heller School for Social Policy and Management, Brandeis University
SALLY MORTON, Graduate School of Public Health, University of Pittsburgh
JOSEPH NEWHOUSE, Division of Health Policy Research and Education, Harvard University
RUTH PETERSON, Criminal Justice Research Center, The Ohio State University
HAL STERN, Department of Statistics, University of California, Irvine
JOHN H. THOMPSON, NORC at the University of Chicago
ROGER TOURANGEAU, Statistical Group, Westat, Rockville, MD
ALAN ZASLAVSKY, Department of Health Care Policy, Harvard Medical School
CONSTANCE F. CITRO, Director
BOARD ON TESTING AND ASSESSMENT
EDWARD HAERTEL (Chair), Jacks Family Professor of Education and Associate Dean for Faculty Affairs, Stanford University
GARY CHAMBERLAIN, Louis Berkman Professor of Economics, Harvard University
MARK DYNARSKI, Researcher, Pemberton Research, LLC
DAVID J. FRANCIS, Hugh Roy and Lillie Cranz Cullen Distinguished Professor, and Director, Texas Institute for Measurement, Statistics and Evaluation, University of Houston
JOAN HERMAN, Director, National Center for Research on Evaluation, Standards, and Student Testing, University of California, Los Angeles
MICHAEL KANE, Messick Chair in Validity, Educational Testing Service
SHARON LEWIS, Director of Research, Council of Great City Schools
ROBERT MARE, Professor of Sociology, University of California, Los Angeles
DIANA C. PULLIN, Professor, Boston College
ANN MARIE RYAN, Professor of Psychology, Michigan State University
BRIAN STECHER, Senior Social Scientist, Education Program, RAND Corporation
JOHN ROBERT WARREN, Associate Professor of Sociology, University of Minnesota
MARK WILSON, Professor of Policy, Organization, Measurement, and Evaluation Cognition and Development, University of California, Berkeley
REBECCA ZWICK, Distinguished Presidential Appointee, Research and Development, Educational Testing Service
STUART ELLIOTT, Director
Acknowledgments
The work of this panel has been immeasurably assisted by the insight and counsel of numerous colleagues. In particular, we wish to acknowledge the reviewers. This report has been reviewed in draft form by individuals chosen for their diverse perspectives and technical expertise, in accordance with procedures approved by the Report Review Committee of the National Research Council (NRC). The purpose of this independent review is to provide candid and critical comments that will assist the institution in making its published report as sound as possible and to ensure that the report meets institutional standards for objectivity, evidence, and responsiveness to the study charge. The review comments and draft manuscript remain confidential to protect the integrity of the deliberative process. We wish to thank the following individuals for their review of this report: Julian Betts, Department of Economics, University of California, San Diego; William G. Bowen, President’s Office, Andrew W. Mellon Foundation; Pat Callan, President’s Office, Higher Education Policy Institute, San Jose, California; Charles T. Clotfelter, Center for the Study of Philanthropy and Voluntarism, Duke University; Don E. Detmer, University of Virginia School of Medicine; David N. Figlio, Institute for Policy Research, Northwestern University; Brent R. Hickman, Economics Department, University of Chicago; Michael McPherson, President’s Office, The Spencer Foundation; B. Don Russell, Jr., Department of Electrical Engineering, Texas A&M University; and Burton A. Weisbrod, Institute for Policy Research, Northwestern University.
The review of this report was overseen by Greg Duncan, distinguished professor of education, University of California, Irvine, and Charles Manski, Board of Trustees professor in economics, Northwestern University. Appointed by the NRC’s Report Review Committee, they were responsible for making certain that
an independent examination of this report was carried out in accordance with institutional procedures and that all review comments were carefully considered.
Although the reviewers have provided many constructive comments, and improved the content of the report a great deal, they were not asked to endorse the conclusions or recommendations; nor did they see the final draft of the report prior to its release. Responsibility for the final content of this report rests entirely with the authoring panel and the institution.
Many others generously gave of their time in offering oral presentations at meetings and answering questions from panel members and staff, thereby helping us to develop a clearer understanding of key issues relevant to the measurement of higher education productivity and related issues. The panel thanks Lumina Foundation; they provided financial support for the project and, even more importantly, helped shape the scope of the study. From Lumina, Jamie Merisotis, Kevin Corcoran, Suzanne Walsh (now with the Bill & Melinda Gates Foundation), and Charles (Chip) Hatcher provided insights and guidance in their roles as initiators of the project. Kristin Conklin (HCM Strategists, LLC) kept the panel informed about Lumina grantees’ programs to increase productivity and proposals for measuring the effectiveness of those efforts. The panel benefited from the open discussion of these initiatives.
During meetings and deliberations, the panel heard from a number of subject matter experts. Andrea Bonaccorsi, University of Pisa, Italy; Kevin Carey, New America Foundation; Hamish Coates, Australian Council for Education Research; Bo Hansson, OECD; Jorge Klor de Alva, University of Phoenix; and Donna Sundre, James Madison University informed the panel about efforts to measure higher education productivity, to design and implement accountability systems, and to improve input/output data at different levels of aggregation.
The panel could not have conducted its work without an excellent and well-managed staff. Connie Citro, director of the Committee on National Statistics, and Stuart Elliott, director of the Board on Testing and Assessment, provided expert guidance to the panel about the NRC study process. Program associate Michael Siri provided excellent administrative, editorial, and research support. Esha Sinha, program officer, provided valuable research and analytic assistance with her understanding of higher education data sources. Her knowledge proved especially helpful as we worked through some of the thornier measurement issues. The panel also benefited from the work of Kirsten Sampson-Snyder, Division of Behavioral and Social Sciences and Education, who was responsible for overseeing the review process. Amy Smith provided able editing of numerous drafts.
Christopher Mackie, the panel’s study director, organized our meetings and facilitated communication among panel members, including a lengthy process of chapter revisions. His work required synthesizing and evaluating many disparate points of input, seeking what common ground could be found, and guiding the panel through careful discussion of the points of disagreement. He helped to
develop the structure for the panel’s final report, and he shepherded the report through the final review process.
Most importantly, the members of the panel deserve thanks for their patience, creativity, and hard work. There is a reason that higher education productivity is not currently reported in the national accounts. Reaching agreement on conceptual and measurement issues was difficult work. This report reflects the collective expertise and commitment of the individual members of the panel, each of whom brought a unique perspective based upon a scholarly discipline, research experience, and a lifetime of practice. Members were generous with their time and effort, and they struggled to understand and appropriately acknowledge the critical views of others. Our meetings provided many opportunities for panel members to learn from one another.
Teresa A. Sullivan, Chair
Panel on Measuring Higher Education Productivity:
Conceptual Framework and Data Needs
Contents
Measurement Limitations and Key Areas for Model Enhancement
Segmentation by Institution Type
Implications of Complexities for Measurement Prospects
Developing the Data Infrastructure
1 THE IMPORTANCE OF MEASURING PRODUCTIVITY IN HIGHER EDUCATION
1.1. Social and Policy Context
1.3. Audience and Report Structure
2 DEFINING PRODUCTIVITY FOR HIGHER EDUCATION
2.1.3. Instructional and Noninstructional Elements of the Higher Education Production Function
2.2. Productivity Contrasted with Other Measurement Objectives
3 WHY MEASUREMENT OF HIGHER EDUCATION PRODUCTIVITY IS DIFFICULT
3.1. Beyond the Degree Factory—Multiple Outputs and Joint Production
3.2. Heterogeneity of Inputs and Outputs
3.3. Nonmarket Variables and Externalities
3.4. Quality Change and Variation
3.5. Measurement at Different Levels of Aggregation
3.5.1. Course and Department Level
4 ADVANCING THE CONCEPTUAL FRAMEWORK
4.2. A Baseline Multi-Factor Productivity Model for Higher Education
4.2.1. Multi-Factor Productivity Indices
4.2.4. Allocations to Education
4.2.5. Illustrative Productivity Calculations
4.3. Institutional Segmentation and Disaggregative Indices
4.3.1. Institutional Segmentation
4.3.2. State-Level and Single-Institution Indices
4.4. Differentiating Labor Categories
4.6. Variations in Output Quality
Technical Appendix: The Törnqvist Productivity Index
5 RECOMMENDATIONS FOR CREATING AND EXTENDING THE MEASUREMENT FRAMEWORK
5.1. The Basic Productivity Measure
5.1.1. Instructional Outputs and Benefits
5.1.2. Instructional Inputs and Costs
5.2. Adjusting for Research Production
5.2.1. Project-Driven Departmental Research
5.2.2. Discretionary Departmental Research
6 IMPLEMENTATION AND DATA RECOMMENDATIONS
6.2. Recommendations for Improving the Data Infrastructure
6.2.1. Data Demanded by the Conceptual Framework
6.2.2. Envisioning the Next Generation IPEDS
6.2.3. Administrative Data Sources
6.2.4. Survey-Based Data Sources
A Commonly Used Performance Metrics for Higher Education