A Guide to the Methodology of the National Research Council Assessment of Doctorate Programs (2009)

Below is the uncorrected machine-read text of this chapter, intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text of each book. Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.

5 An Example Shortly before the assessment is released, each institutional coordinator will receive three tables for each program that was ranked. These will reflect the following: (1) the values that they submitted or were calculated from their data for each of the 20 variables with their corresponding standardized values, and (2) a pair of combined coefficients (plus and minus one standard deviation from the average value) used in weighting the variables (see Table 5-1); and (3) the standardized program values and the actual combined coefficients that were used to calculate the rating corresponding to each endpoint of the inter-quartile range of rankings for that program, as well as the program ranking corresponding to those ratings (see Tables 5-2a and 5- 2b). Examples of these tables for an economics program are presented and discussed below. Table 5-1 shows the values submitted by an unidentified program in economics and the range of combined coefficients for the entire field. Columns 1 and 2 name and label the variables. Column 3 gives the program value for each of the 20 variables used in the overall rating (see Appendix E for a description of these variables). Column 4 presents the standardized value of each variable in column 3; scores are standardized across all programs in the field, using a mean of 0 and variance of 1. Thus, the relative strengths and weaknesses of a program (in terms of these 20 variables) can be seen by comparing the standardized values in column 4. Columns 5 and 6 give the pairs of combined coefficients (weights) assigned to each variable used in rating all economics programs.29 Each coefficient is a combination of both the direct and regression-based weights, the derivation of which is described in detail in Appendix A. In economics, variables V1, V2, and V14 (publications per allocated faculty, cites per publication and average number of Ph.D.’s) were assigned the largest weights. Although it would be relatively easy to calculate a single rating for the program using the data in Table 5-1, the result could be misleading, because it would not reflect the variability (i.e., uncertainty) in each of the program measures or the variability in the estimation of the weights. The process for taking into account these sources of variability is described in detail in Appendix A. 29 Five hundred regressions are run using half of the raters each time and 500 draws are made from randomly selected halves of the pool of direct ratings in order to construct the combined coefficients. The values presented show the range encompassed by plus or minus one standard deviation for each coefficient. See Appendix A for details. 19 PREPUBLICATION COPY—UNEDITED PROOFS 

TABLE 5-1 Data and Coefficient Table for a Program in Economics Standardized Program Values and Range of Combined Coefficients Institution Name: xxx Program Name: yyy Column 1 Column 2 Column 3 Column 4 Column 5 Column 6 Combined Coefficients** Program Program Value Description Variable Value* Standardized* Minus 1 SD Plus 1 SD Publications per Allocated Faculty V1 1.074 2.180 0.118 to 0.132 Cites per Publication V2 1.171 -0.234 0.276 to 0.307 Percent of Faculty with Grants V3 25.50% -0.583 0.084 to 0.091 Percent Faculty Interdisciplinary V4 5.90% -0.641 n.s.# n.s.# Percent Non-Asian Minority Faculty V5 7.70% 0.547 n.s.# n.s.# Percent Female Faculty V6 12.50% -0.440 n.s.# n.s.# Awards per allocated faculty V7 0 -0.546 0.043 to 0.060 Average GRE-Q V8 746 -0.165 0.092 to 0.096 Percent 1st yr. students w/ full support V9 100.00% 0.980 0.036 to 0.056 Percent 1st yr students with portable fellowships V10 0.00% -0.544 0.021 to 0.033 Percent Non-Asian Minority Students V11 10.00% 0.069 n.s.# n.s.# Percent Female Students V12 44.40% 0.678 -0.038 to -0.030 # Percent International Students V13 53.30% -0.509 n.s. n.s.# Average PhDs 2002 to 2006 V14 5.4 -0.355 0.120 to 0.144 Percent Completing within 6 years V15 27.60% -0.638 n.s.# n.s.# Time to Degree Full and Part Time V16 5.67 0.232 -0.028 to -0.017 Percent students in Academic Positions V17 11.10% -1.405 0.049 to 0.065 Student Work Space V18 1 1 n.s.# n.s.# # Health Insurance V19 1 1 n.s. n.s.# Number of student activities offered V20 17 0.439 0.026 to 0.037 *Col 3 is based on data submitted by the program or calculated from these data. + Col 4 is standardized across all program values in the field, with mean of 0 and variance of 1. ** Col 5 is Minus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole ** Col 6 is Plus 1 Standard Deviation from the Mean for the combined coefficients for the field as a whole # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. 20 PREPUBLICATION COPY—UNEDITED PROOFS 

Tables 5-2a and 5-2b show the calculations of the first and third quartile rankings, respectively, for a particular program.30 First, a randomly sampled set of regression coefficients and direct weights is used to obtain a set of 20 combined weights (column 5). These weights are multiplied by a sampled set of standardized program values (column 4) to generate a program rating (sum of column 6). This process is repeated another 499 times, generating 500 ratings for each of the 117 economics programs. Each of these 500 ratings for the program is ranked by comparing it with the ratings for the other 116 economics programs, based on the same selection of weights. The 500 rankings for the program are then ordered from best to worst, with the 125th being the Quartile 3 ranking (45) and the 375th being the Quartile 1 ranking (56). These values determine the inter-quartile range of rankings for the program. Half of the 500 randomly generated rankings for the program fall within this range31. The ratings that produced these first and third quartile rankings are -0.054 and 0.085, as shown in Tables 5-2a and 5-2b.32 30 The first quartile ranking is the highest value of the lowest quarter of rankings. The third quartile ranking is the highest value of the third quarter of rankings. 31 Use of the inter-quartile range means that we “throw away” half of the possible rankings for the program. The tails of the distribution can be very long, however, and the inter-quartile range is useful in making meaningful comparisons, while illustrating the point that any point estimate of a ranking is inexact. 32 We do not show the 117 x 500 matrix of all the ordered ratings for all the economics programs, although it will be available when the final report is released. However, the ranking is obtained from that table. 21 PREPUBLICATION COPY—UNEDITED PROOFS 

Table 5-2a Sample First Quartile Ranking Calculation Institution Name: xxx Program Name: yyy Col 1 Col 2 Col 3 Col 4 Col 5 Col 6 Standardized Program Program Value Combined Product Col Description Variable Value* with Variation+ Coefficient@ 4 x Col 5 Publications per Allocated Faculty V1 1.074 1.784 0.130 0.231 Cites per Publication V2 1.171 -0.269 0.294 -0.079 Percent of Faculty with Grants V3 25.5% -0.596 0.085 -0.051 Percent Faculty Interdisciplinary V4 5.9% -0.581 n.s. # n.c. # Percent Non-Asian Minority Faculty V5 7.7% 0.444 n.s. # n.c. # Percent Female Faculty V6 12.5% -0.511 n.s. # n.c. # Awards per allocated faculty V7 0 -0.290 0.038 -0.011 Average GRE-Q V8 746 -0.286 0.091 -0.026 Percent 1st yr. students w/ full support V9 100% 1.432 0.044 0.064 Percent 1st yr students with portable fellowships V10 0.0% -0.489 0.023 -0.011 Percent Non-Asian Minority Students V11 10.0% 0.062 n.s. # n.c. # Percent Female Students V12 44.4% 0.561 -0.029 -0.016 # Percent International Students V13 53.3% -0.018 n.s. n.c. # Average PhDs 2002 to 2006 V14 5.4 -0.379 0.152 -0.058 # Percent Completing within 6 years V15 27.6% -0.574 n.s. n.c. # Time to Degree Full and Part Time V16 5.67 0.017 -0.026 0.000 Percent students in Academic Positions V17 11.1% -1.365 0.063 -0.086 Student Work Space V18 1 1.000 n.s. # n.c. # Health Insurance V19 1 1.000 n.s. # n.c. # Number of student activities offered V20 17 -0.427 0.025 -0.011 Average Rating (total of column 6) -0.054 Program Ranking for this rating = 56 *Col 3 is based on data submitted by the program or calculated from these data. +Col 4 is standardized value for the set of perturbed program values that produced the 1st quartile ranking. Standardized values have a mean of 0 and variance of 1. @ Col 5 is the combined direct and regression-based weights for each variable (see Appendix A). # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. # n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level. 22 PREPUBLICATION COPY—UNEDITED PROOFS 

Table 5-2b Sample Third Quartile Ranking Calculation Institution Name: xxx Program Name: yyy Col 1 Col 2 Col 3 Col 4 Col 5 Col 6 Standardized Program Program Value with Combined Product Description Variable Value* Variation+ Coefficient@ Col 4 x Col 5 Publications per Allocated Faculty V1 1.074 2.765 0.134 0.371 Cites per Publication V2 1.171 -0.246 0.267 -0.066 Percent of Faculty with Grants V3 25.5% -0.709 0.073 -0.051 Percent Faculty Interdisciplinary V4 5.9% -0.669 n.s. # n.c. # Percent Non-Asian Minority Faculty V5 7.7% 0.515 n.s. # n.c. # Percent Female Faculty V6 12.5% -0.314 n.s. # n.c. # Awards per allocated faculty V7 0 -0.439 0.050 -0.022 Average GRE-Q V8 746 -0.305 0.089 -0.027 Percent 1st yr. students w/ full support V9 100% 0.385 0.054 0.021 Percent 1st yr students with portable fellowships V10 0.0% -0.585 0.031 -0.018 Percent Non-Asian Minority Students V11 10.0% 0.226 n.s. # n.c. # Percent Female Students V12 44.4% 0.083 -0.043 -0.004 # Percent International Students V13 53.3% -0.190 n.s. n.c. # Average PhDs 2002 to 2006 V14 5.4 -0.196 0.121 -0.024 # Percent Completing within 6 years V15 27.6% -0.725 n.s. n.c. # Time to Degree Full and Part Time V16 5.67 -0.439 -0.031 0.014 Percent students in Academic Positions V17 11.1% -1.293 0.083 -0.108 Student Work Space V18 1 1.000 n.s. # n.c. # Health Insurance V19 1 1.000 n.s. # n.c. # Number of student activities offered V20 17 -0.058 0.024 -0.001 Average Rating (total of column 6) 0.085 Program Ranking for this rating = 45 *Col 3 is based on data submitted by the program or calculated from these data. +Col 4 is standardized value for the set of perturbed program values that produced the 3rd quartile ranking. Standardized values have a mean of 0 and variance of 1. @ Col 5 is the combined direct and regression-based weights for each variable (see Appendix A). # n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level. # n.c. means the product was not calculated for these coefficients because the coefficient was not significant at the p=.05 level. 23 PREPUBLICATION COPY—UNEDITED PROOFS 

In interpreting the range of rankings a program received, the first thing to note is which variables have the highest coefficients. These variables can be determined by examining the combined coefficients and identifying the largest ones. In the case of economics, the important variables are citations per publication, publications per allocated faculty, average Ph.D.’s in 2002-2006, and average GRE-Q, each of which has a combined coefficient value of 0.089 or greater. The rest of the variables are less heavily weighted, and a number of the variables don’t enter into the determination of the overall rating at all because their coefficients were not statistically different from 033. The program values in column 3 of Table 1 can be contrasted with the values taken across all the values in the field, shown in Table 5-3. The importance of correcting for collinearity 34 is evident from the correlation matrix that follows the variable listing for each field, and is shown in Table 5-4. Citations per publication, for example, have a correlation .7 with awards, and .5 with GRE-Q, with average Ph.D.’s and with percent completing within six years. This interdependence is corrected for by the principal components adjustment described in Appendix A. 33 The procedure for setting nonsignificant coefficients to 0 is discussed in Appendix A. 34 That is, high degrees of correlation among some of the independent variables. 24 PREPUBLICATION COPY—UNEDITED PROOFS 

TABLE 5-3 Descriptive Statistics for the Variables used in the Ratings: All Economics Programs 1st 3rd Standard Minimum Quartile Quartile Maximum Deviation Publications per Allocated Faculty 0.049 0.369 0.655 1.357 0.246 Cites per Publication 0.153 0.684 1.771 5.485 1.002 Percent of Faculty with Grants 0.0% 24.0% 50.0% 100.0% 19.9% Percent Faculty Interdisciplinary 0.0% 2.1% 26.9% 68.4% 16.3% Percent Non-Asian Minority Faculty 0.0% 0.0% 7.7% 25.0% 5.3% Percent Female Faculty 0.0% 10.5% 21.1% 66.7% 9.9% Awards per allocated faculty 0.000 0.000 0.462 5.131 0.890 Average GRE-Q 353 740 790 800 55 Percent 1st yr. students w/ full support 0.0% 50.0% 100.0% 100.0% 29.2% Percent 1st yr students with portable fellowships 0.0% 0.0% 9.1% 66.7% 14.2% Percent Non-Asian Minority Students 0.0% 0.0% 14.3% 50.0% 9.8% Percent Female Students 0.0% 28.6% 42.9% 76.9% 12.0% Percent International Students 0.0% 52.4% 76.3% 98.2% 19.6% Average Ph.D.s 2002 to 2006 1.00 3.20 9.80 26.40 5.73 Percent Completing within 6 years 0.0% 28.3% 51.0% 91.7% 19.1% Time to Degree Full and Part Time 3.00 5.00 6.00 8.00 0.80 Percent students in Academic Positions 4.2% 17.6% 39.6% 56.5% 12.5% Student Work Space -1 -1 1 1 0.985 Health Insurance -1 1 1 1 0.672 Number of student activities offered 4 15 18 18 2.161 25 PREPUBLICATION COPY—UNEDITED PROOFS 

Highest Lowest Highest Lowest Quartile Quartile Highest Lowest Highest Lowest Quartile Quartile Student Student Quartile Quartile Quartile Quartile Table 5-4 Correlations Matrix- Research Research Support Support Diversity Diversity Overall Overall Economics Rating Rating Rating Rating Rating Rating Rating Rating 3rd Quartile Research Rating 1.00 1st Quartile Research Rating 1.00 1.00 3rd Quartile Student Support Rating 0.39 0.39 1.00 1st Quartile Student Support Rating 0.38 0.38 1.00 1.00 3rd Quartile Diversity Rating -0.24 -0.25 -0.24 -0.23 1.00 1st Quartile Diversity Rating -0.25 -0.25 -0.24 -0.24 1.00 1.00 3rd Quartile Overall Rating 0.95 0.94 0.45 0.44 -0.23 -0.24 1.00 1st Quartile Overall Rating 0.95 0.95 0.44 0.43 -0.23 -0.23 1.00 1.00 Publications per Allocated Faculty 0.78 0.79 0.32 0.31 -0.23 -0.24 0.64 0.64 Cites per Publication 0.86 0.84 0.33 0.32 -0.16 -0.17 0.90 0.90 Percent of Faculty with Grants 0.55 0.56 0.09 0.08 -0.12 -0.12 0.51 0.52 Percent Faculty Interdisciplinary 0.05 0.05 0.14 0.14 -0.12 -0.13 0.10 0.11 Percent Non-Asian Minority Faculty 0.03 0.03 0.10 0.10 0.39 0.37 0.00 0.00 Percent Female Faculty -0.19 -0.20 -0.26 -0.26 0.45 0.45 -0.21 -0.21 Awards per allocated faculty 0.77 0.76 0.41 0.40 -0.21 -0.22 0.77 0.77 Average GRE-Q 0.52 0.51 0.23 0.22 -0.08 -0.08 0.66 0.65 Percent 1st yr. students w/ full support 0.21 0.21 0.58 0.58 0.00 -0.01 0.24 0.24 Percent 1st yr students with portable fellowships 0.34 0.34 0.21 0.22 -0.13 -0.12 0.37 0.37 Percent Non-Asian Minority Students -0.12 -0.12 0.02 0.02 0.49 0.47 -0.13 -0.13 Percent Female Students -0.29 -0.28 -0.33 -0.33 0.40 0.41 -0.35 -0.34 Percent International Students -0.10 -0.10 -0.14 -0.14 0.67 0.69 -0.03 -0.03 Average Ph.D.s 2002 to 2006 0.58 0.57 0.04 0.03 -0.11 -0.11 0.70 0.70 Percent Completing within 6 years 0.50 0.50 0.57 0.56 -0.32 -0.32 0.54 0.54 Time to Degree Full and Part Time -0.14 -0.16 -0.36 -0.34 -0.07 -0.07 -0.10 -0.10 Percent students in Academic Positions 0.20 0.20 0.71 0.73 -0.20 -0.20 0.27 0.27 26 PREPUBLICATION COPY—UNEDITED PROOFS 

Student Work Space 0.13 0.15 0.28 0.28 -0.11 -0.11 0.12 0.12 Health Insurance 0.23 0.23 0.19 0.17 -0.11 -0.12 0.25 0.25 Number of student activities offered 0.05 0.04 0.01 0.01 0.04 0.03 0.16 0.15 Percent Percent Percent 1st yr Publications Percent of Non- Awards 1st yr. students Correlations- Economics cont'd (2) per Faculty Asian Percent per students with Allocated Cites per with Percent Faculty Minority Female allocated Average w/ full portable Faculty Publication Grants Interdisciplinary Faculty Faculty faculty GRE-Q support fellowships 3rd Quartile Research Rating 1st Quartile Research Rating 3rd Quartile Student Support Rating 1st Quartile Student Support Rating 3rd Quartile Diversity Rating 1st Quartile Diversity Rating 3rd Quartile Overall Rating 1st Quartile Overall Rating Publications per Allocated Faculty 1.00 Cites per Publication 0.40 1.00 Percent of Faculty with Grants 0.36 0.37 1.00 Percent Faculty Interdisciplinary -0.08 0.08 0.04 1.00 Percent Non-Asian Minority Faculty 0.06 -0.02 0.12 -0.02 1.00 Percent Female Faculty -0.29 -0.06 0.01 -0.08 0.10 1.00 Awards per allocated faculty 0.42 0.70 0.28 0.20 -0.03 -0.21 1.00 Average GRE-Q 0.37 0.52 0.16 0.10 -0.19 -0.26 0.38 1.00 Percent 1st yr. students w/ full support 0.20 0.16 0.06 0.08 0.17 -0.17 0.20 0.23 1.00 Percent 1st yr students with portable fellowships 0.13 0.36 0.14 0.05 -0.14 -0.19 0.43 0.14 0.12 1.00 Percent Non-Asian Minority Students -0.09 -0.11 -0.18 0.01 0.42 0.07 0.01 -0.21 -0.01 0.04 Percent Female Students -0.24 -0.22 -0.06 0.00 0.05 0.18 -0.30 -0.15 -0.04 -0.08 27 PREPUBLICATION COPY—UNEDITED PROOFS 

Percent International Students -0.09 -0.06 -0.10 -0.14 -0.10 0.06 -0.09 0.25 0.02 -0.03 Average Ph.D.s 2002 to 2006 0.33 0.53 0.38 0.11 -0.04 -0.08 0.55 0.44 -0.07 0.20 Percent Completing within 6 years 0.31 0.52 0.09 0.10 -0.17 -0.13 0.52 0.34 0.24 0.28 Time to Degree Full and Part Time -0.19 -0.09 -0.06 0.06 -0.12 -0.03 -0.06 0.04 -0.05 -0.03 Percent students in Academic Positions 0.15 0.15 0.03 0.14 0.13 -0.27 0.28 0.10 0.11 0.11 Student Work Space 0.17 0.12 0.12 -0.02 0.04 -0.13 -0.06 0.12 0.23 0.06 Health Insurance 0.19 0.21 0.13 0.03 0.10 -0.12 0.12 0.22 0.17 -0.21 Number of student activities offered -0.07 0.12 0.12 0.12 0.15 -0.04 0.00 0.13 -0.05 -0.08 28 PREPUBLICATION COPY—UNEDITED PROOFS 

Time to Percent Percent Average Percent Degree students Number Correlations- Economics cont'd (3) Non-Asian Percent Percent Ph.D.s Completing Full and in Student of student Minority Female International 2002 to within 6 Part Academic Work Health activities Students Students Students 2006 years Time Positions Space Insurance offered 3rd Quartile Research Rating 1st Quartile Research Rating 3rd Quartile Student Support Rating 1st Quartile Student Support Rating 3rd Quartile Diversity Rating 1st Quartile Diversity Rating 3rd Quartile Overall Rating 1st Quartile Overall Rating Publications per Allocated Faculty Cites per Publication Percent of Faculty with Grants Percent Faculty Interdisciplinary Percent Non-Asian Minority Faculty Percent Female Faculty Awards per allocated faculty Average GRE-Q Percent 1st yr. students w/ full support Percent 1st yr students with portable fellowships Percent Non-Asian Minority Students 1.00 Percent Female Students 0.05 1.00 Percent International Students -0.04 0.05 1.00 Average Ph.D.s 2002 to 2006 -0.10 -0.27 0.06 1.00 Percent Completing within 6 years -0.13 -0.36 -0.13 0.28 1.00 Time to Degree Full and Part Time -0.08 0.01 -0.01 0.10 -0.33 1.00 Percent students in Academic Positions 0.11 -0.25 -0.18 0.03 0.22 -0.03 1.00 29 PREPUBLICATION COPY—UNEDITED PROOFS 

Student Work Space -0.04 -0.04 -0.08 -0.15 0.26 -0.26 0.09 1.00 Health Insurance -0.17 -0.11 -0.01 0.16 0.06 0.01 0.10 -0.05 1.00 Number of student activities offered 0.02 -0.11 0.05 0.16 -0.02 0.09 0.10 -0.05 0.19 1.00 30 PREPUBLICATION COPY—UNEDITED PROOFS 

The overall range of rankings should be looked at in the context of the dimensional measures for economics shown in Appendix G. Typically, programs that score well on the overall rankings will also do well on the research activity ranking, because the two have a number of highly weighted components in common. It is also worthwhile to look at the student support and outcomes ranking and the diversity ranking, because these may be of importance to students in selecting a program. The economics program’s overall measure—it is program number 62 in the table in Appendix G—places it between the 45th and 56th of the 117 programs. Looking at the dimensional rankings, its research activity is highly ranked—between the 21st and 31st—primarily because of a relatively high rate of publications per allocated faculty member. It does less well in terms of student support and outcomes, where it ranks between the 74th and 87th. Nor does it perform especially well on the diversity dimensional measure—its rank is between the 64th and 77th. The dimensional measures, then, indicate the specific areas in which programs are performing well or poorly, as separate from the overall range of rankings. The example is intended to explain to the reader how ratings are calculated, and how a range of rankings is constructed. Shortly before the study results are released, each institutional coordinator will receive tables similar to the tables above, showing the program data, the range of coefficients for each variable, the calculation of the first and third quartile rating, and the corresponding ranking for each rated program at the institution. The user should be aware, however, that he or she cannot duplicate all 500 samples of combined coefficients. After the report is released, software will be provided that will permit simulations of ratings with user- supplied weights and alternative data values. Because the ratings depend on program data and weights, both of which have uncertainties associated with them, the ranking resulting from a simulation can only be approximate. The committee would advise that the calculations are more useful in a qualitative sense. That is, for the numerous programs that fall in the middle range of rankings, it doesn’t make sense to focus on an exact range. It does make sense to identify the variables that are important to the ranking of each program and, where possible, improve them35. 35 An example would be working to shorten time to degree. 31 PREPUBLICATION COPY—UNEDITED PROOFS 

32 PREPUBLICATION COPY—UNEDITED PROOFS