National Academies Press: OpenBook

A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs (2009)

Chapter: 3 Ratings in Specific Dimensions: The Dimensional Measures

« Previous: 2 The Data and How They Were Obtained
Suggested Citation:"3 Ratings in Specific Dimensions: The Dimensional Measures." National Research Council. 2009. A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs. Washington, DC: The National Academies Press. doi: 10.17226/12676.
×
Page 11
Suggested Citation:"3 Ratings in Specific Dimensions: The Dimensional Measures." National Research Council. 2009. A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs. Washington, DC: The National Academies Press. doi: 10.17226/12676.
×
Page 12
Suggested Citation:"3 Ratings in Specific Dimensions: The Dimensional Measures." National Research Council. 2009. A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs. Washington, DC: The National Academies Press. doi: 10.17226/12676.
×
Page 13

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

3 Ratings in Specific Dimensions: The Dimensional Measures The dimensional measures are provided to assure that measures of a broad range of characteristics of doctoral programs are available. They are divided into three categories: (1) research activity, (2) student support and outcomes, and (3) diversity of the academic environment. Each of the dimensional measures begins with the measures relating to one dimension of doctoral program performance, applies the weights from the faculty survey about what program characteristics contribute to quality, and then constructs a range17 of rankings for each program based on this dimension of the data, taking into account variability in the data and in the choice of raters. They are dimensional in the sense that they provide more focused measures than the overall range of rankings, but they are central to the calculation of this range. Some specifics about the calculation of these measures follow. • How the weights are obtained—As part of the NRC faculty questionnaire, we asked faculty to indicate the relative importance of different characteristics of doctoral programs; this was done through the multipart question that makes up Section G of the faculty questionnaire (see Appendix B). Faculty were questioned about faculty quality, student characteristics, and program characteristics. First they were asked to indicate up to four characteristics in each category that they thought were important to program quality. Each characteristic that was listed received an initial score of 1. These preferences were then narrowed by asking the faculty members to identify a maximum of two characteristics in each category that they thought were most important. These characteristics each received a score of 2. A final question asked faculty members to indicate the relative importance of each category by assigning category weights whose values summed to 100. For each individual faculty member, the weight for a variable was calculated as the sum of the “votes” that it received times the importance assigned to the category that contained it. The weight for a variable in a discipline was the average weight taken across all faculty members in it. We 17 When we use the term “range,” we are referring to the inter-quartile range. This is the range that contains half of the observations or estimates of the quantity of interest. 11 PREPUBLICATION COPY—UNEDITED PROOFS

took into account variability in raters’ opinions and uncertainties due to missing data and the fact that some measures were sampled at one point in time.18 Approximately 86 percent of the faculty responded. Their responses permitted calculation of the set of “direct” weights. Although there was some variation in the faculty responses, they were generally in agreement that publications and citations were the most important factors in program quality. Every variable, however, received some weight19. These weights were used to construct the dimensional measures. The average weights for programs in each broad field are shown in Appendix F, and an example of ranges of rankings for programs in economics is shown in Appendix G. • Research activity—This dimensional measure relates to various ways to gauge the contribution of research: publications, citations (except for the humanities), the percent of the faculty holding research grants, and recognition of scholarship as evidenced by honors and awards. Specifically, the components of the research activity dimensional measure are: average publications per allocated faculty member,20 average citations per publication, percent of core and new doctoral faculty respondents holding grants, and awards per allocated faculty member. Publishing patterns and the availability of research funding and awards for scholarship vary by field, but the weight placed on publications per faculty member is remarkably consistent—about 30 percent—across fields. Research activity is the dimensional measure that most closely tracks the overall measure of program quality, because in all fields, both the direct measure—based on abstract faculty preferences—and the regression-based measure put high weight on these measures. • Student support and outcomes—This measure combines data on the percent of students fully funded in the first year, the percent of students completing their degrees in a given time period, time to degree, placement in academic positions (including academic postdoctoral positions), and whether a program collects data about the employment outcomes for its students. We found that faculty typically placed a larger weight on student support and completion rates than on median time to degree, academic placement, or whether a program 18 There is some uncertainty in the values of the program variable values themselves. Some of the 20 program variables used to calculate the ratings also vary or have an error associated with their values due to year-to-year fluctuations. Data for five of the variables (publications per faculty, citations per publications, GRE scores, Ph.D. completion, and number of Ph.D.’s) were collected over time, and averages over a number of years were used as the values of these program variables. If a different time period had been used, the values would have been different. To express this type of uncertainty, a relative error term, ejk, was associated with each variable value. For details, see Appendix A. 19 All “direct” weights are used in the calculation of the Dimensional Measures. This differs from the case of the Overall Rating of Program Quality, in which some coefficients might be set to zero if the result of combining the direct and the regression-based weights was not statistically significantly different from zero. 20 Because many faculty members supervise dissertations in more than one program, faculty members were allocated across these programs so that the total, taken across all programs, equaled one or less (in the case in which the faculty member was in a professional school). 12 PREPUBLICATION COPY—UNEDITED PROOFS

follows the employment outcomes of it students.21 There is surprising uniformity across broad fields on the weights, which are shown in Appendix F. • Diversity of the academic environment—The diversity measures did not appear as major factors in determining the overall perceived quality of programs. Taken separately, there are definite patterns for variables that faculty thought were more important, and these vary by field. The measures that are included in this dimensional measure are: the percent of faculty and percent of students who are from underrepresented minority groups, the percent of faculty and the percent of students who are female, and the percent of students who are international (that is, in the United States on a temporary visa). In terms of field differences, most fields place the highest weight on the percentage of students from underrepresented minority groups. In the health sciences, social sciences, and humanities, relatively high weights are also placed on the percentage of faculty who are underrepresented minorities. The percentage of international students was not highly weighted, except for the physical sciences. These weights, by broad field, are shown in Appendix F. What is interesting about the dimensional ratings is that, with the exception of the research activity measure, they produce program rankings that are quite different from the overall ratings. This can be seen in the table in Appendix G. Excellence in doctoral programs is not uni-dimensional. Some students may prefer a program where they can be assured of steady funding and a short time to degree, even if it is not a program that is perceived as stellar in terms of the productivity of its faculty. Similarly, a program that is more diverse may be preferable to many students, although diversity bears only a tenuous relation with the usual measures of scholarly productivity. Users of the assessment should be aware of these different dimensions, because each presents the characteristics of an individual doctoral program from a different perspective. 21 Ideally, we would have used a measure such as employment in one’s field 5 years after receipt of Ph.D., but many programs did not collect such data. The committee hoped that including this measure would encourage more programs to pay attention to post-degree outcomes for their graduates 13 PREPUBLICATION COPY—UNEDITED PROOFS

Next: 4 The Overall Rating of Program Quality »
A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs Get This Book
×
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

A Guide to the Methodology of the National Research Council Assessment of the Doctorate Programs describes the purpose, data and methods used to calculate ranges or rankings for research-doctorate programs that participated in the NRC Assessment of Research-Doctorate programs. It is intended for those at universities who will have to explain the NRC Assessment to others at their university, to potential students, and to the press. Although the main text is fairly non-technical, it includes a technical description of the statistical methods used to derive rankings of over 5000 doctoral programs in 61 fields.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  6. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  7. ×

    View our suggested citation for this chapter.

    « Back Next »
  8. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!