|
Items in cart [1] |
Questions? Call 888-624-8373 |
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 185
APPENDIX
A
A Review of the Methodology Used
in the Integrated Quality Control
Measurement Project
Mark Reiser
Arizona State University
This paper is a review of the methodology used in the Integrated Qual-
ity Control Measurement Project (IQCMP), a project conducted to evaluate
the quality of financial awards made in the 1988-89 award year under three
Title IV financial aid programs: the Pell Grant Program, Campus-Based
Programs, and the Stafford Loan Program (see Price Waterhouse, 1990~. In
addition to comments on the methodology that was used in the IQCMP,
possible alternative methodologies are suggested.
ERROR
The IQCMP used the following definition of error: "Error is the differ-
ence between the award actually distributed and the award that would be
calculated based on the best available data." Data that were considered to
be the best available were obtained from the source of highest reliability
(e.g., income tax returns and interviewers). One source was defined to be
the most reliable, and it was then used as though it contained no error.
When one source of information is treated as though it contains no error, it
is sometimes referred to as the gold standard. The difference in awards as
defined above had to exceed $50.00 before it was considered to be an error.
In the IQCMP, the student and the academic institution were examined
as sources of error in each of the Title IV programs studied. Overall and
composite error were also examined. Overall error was error resulting
from either student or institutional error. Student error and institutional
185
OCR for page 186
86
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
error do not necessarily add to overall error, however, because they do not
necessarily have an additive effect.
Composite error is error in the financial award from all programs com-
bined. For this type of error, two main types of analysis were performed.
First, the magnitude and percentage of errors were examined. Then, in
error-profile analyses, the presence or absence of error was modeled using
several independent variables simultaneously.
As noted above, error calculations were performed using one or another
source of information as an error-free gold standard. Since these sources
used as standards almost certainly contain error themselves, the calculation
of errors itself contains error. This feature constitutes a source of nonsampling
error in the IQCMP. When no single source of information can be com-
pletely free of error, adopting a statistical model for measurement error is a
better approach, because the true value is considered to be a latent (unob-
servable) variable and the different sources of measurement, called indica-
tor or manifest variables, are all acknowledged as being subject to error. In
many areas of social science, the reliability (or reliability ratio) is used as
an indicator of the degree to which a variable is free from measurement
error. In a technical definition, the reliability is expressed as the ratio of
true variance to total variance, where total variance is equal to true variance
plus measurement-error variance. Since no single source of information for
determining financial aid eligibility is absolutely error free, it would have
been very useful if the IQCMP had analyzed and reported the reliability of
the different data sources. If more than one indicator is available, the
reliabilities may be calculated in a straightforward way. The true value can
also be estimated from a measurement-error model for multiple indicators.
In future studies of quality control in student financial aid programs, the
estimated true value for variables, such as wealth, could be calculated and
used to determine the error in the amount of the financial award.
Generally, variables with low reliability are not very useful for calculat-
ing eligibility for financial awards. In the past, it appears that variables
with large measurement error (low reliability) were either dropped from use
on the student financial aid application or were the focus of regulations
intended to decrease error during data collection. To the extent that error
can be reduced with reasonable efforts, it is a good strategy to try to do so.
But for some variables, such as wealth, it is very difficult to measure the
true value directly, and an alternative to dropping such variables would be
to estimate the latent true value from multiple sources of information, each
of which may not be very reliable by itself.
Measurement-error methodology should be investigated in any future
studies of quality control in student financial aid programs. It is an ap-
proach that is more efficient and justifiable than declaring one of the indica-
tors to be the gold standard.
OCR for page 187
APPENDIX A
187
SAMPLING
The IQCMP was based on a sample of 3,310 students from 350 postsecondary
institutions. Because data were not available for some students, 2,653 stu-
dents were in the final data base. Details of the sampling procedure are not
given in the IQCMP report, but they are provided in a separate sampling
plan document (Price Waterhouse, 1989~. A two-stage cluster sample was
used in which postsecondary institutions were sampled in clusters of size 1,
2, 3, or 4. Because observations within each cluster are correlated to an
unknown extent, methods of analysis that assume simple random samples
are not appropriate. For proper analysis, each element of the sample must
be assigned a weight based on the probability of selection for that element.
In the sampling plan, expressions for the probability of selection are given
on pages IV-8 and IV-9. The expressions appear to be incorrect, however,
because they give the probability of selection as a sum of sampling rates
across programs. The probability of selection calculated from this expres-
sion could be greater than 1.O, although the sampling rates actually used in
the study are probably not high enough for that to happen. The correct
expression for the probability of selection for an element, given the sam-
pling plan, should be
1.0 - (probability not selected for Pell sample) x
(probability not selected for Campus-Based sample) x
(probability not selected for Stafford sample).
The probability calculated from this expression applies to the selection for
the sample of all financial aid recipients. It is not the probability of selec-
tion for an element in any combined sample (i.e., a direct sample plus
elements obtained as a by-product of sampling other programs). Moreover,
it is not clear how the probability of selection for an element in a combined
sample should be calculated, because it would require knowing the prob-
ability that a recipient is in one program given that he or she is in another
program.
If probabilities of selection were calculated by the expressions given on
pages IV-8 and IV-9 of the sampling plan, the weights used for calculating
errors would have been incorrect, and estimates of error would be biased. It
is not possible to say how large the bias would be: It could be trivial or it
could be substantial.
In addition to sampling clusters of schools, the efficiency of the sam-
pling design could have been increased through stratification by a number
of variables, such as financial dependence versus independence. Since strati-
fication was not used, post-stratification could be used to increase the preci-
sion of some estimates.
OCR for page 188
88
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
As stated in the IQCMP report (p. I-2), the sample of students was
designed to be large enough to permit statistical inferences about the per-
centage of awards in error for each Title IV program with + 5 percent
accuracy at a 90 percent confidence level. This statement was apparently
made with complex sampling in mind, since a simple random sample of
2,653 would allow statistical inference with + 1.6 percent accuracy at a 90
percent confidence level. However, as discussed next, standard errors are
not used consistently in the IQCMP report.
The exhibits in Section II ("Error in the Title IV Financial Aid Pro-
grams") of the IQCMP report present standard errors for many estimates.
The basis for the standard errors is not clear, however. The standard errors
given in the Section II exhibits have magnitudes (0.4 to 2.5) that appear to
be based on an assumption of a simple random sample rather than the
complex sample that was actually used. Although discussion in the sam-
pling plan indicates that some standard errors were calculated by a boot-
strap method, some may have been calculated by assuming a stratified ran-
dom sample and ignoring clustering. It is not clear why standard errors
would be calculated that way if bootstrap methods were available. More-
over, estimators for standard errors under two-stage cluster sampling are
known (Cochran, 1977), and computer programs are widely available to
perform the calculations (e.g., Super Carp). The description of the boot-
strap method proposed in the sampling plan is very brief, and it is not clear
how the bootstrap was implemented, particularly with regard to the selec
~ . .
lion of primary sampling units.
Only Section II of the IQCMP report gives standard errors. Other
sections of the report give a large number of statistics, but standard errors
are not given.
A footnote to Exhibit II-1 states that the percentage of recipients with
error is based on 6.0 million students awarded Title IV aid during the 1988-
89 award year. Similar footnotes can be found in other Section II exhibits.
The purpose of the footnotes appears to be to emphasize that the errors were
examined only among the recipients of Title IV financial aid, and not among
students who applied for but were denied financial aid. Nevertheless, the
footnotes are confusing and should be reworded, because they can be inter-
preted to mean that the percentages are based on the entire population of 6
million recipients rather than a sample.
Many sections of the IQCMP report compare percentages across vari-
ables and across domains. Financially dependent versus independent stu-
dents and proprietary versus nonproprietary schools are frequently men-
tioned domains. The statement in Section II regarding + 5 percent accuracy
has very limited applicability for comparisons across variables or domains,
and generally the appropriate interval would be larger than + 5 percent. For
comparisons across domains, one would probably be willing to assume in
OCR for page 189
APPENDIX A
189
dependent samples for the different domains, although that would not nec-
essarily be true, since both financially dependent and independent students
would be selected from each school, and the clustering effect would induce
a correlation among the students from the same school. However, assuming
samples are independent across domains and that the domains split the
sample into roughly equal-sized groups, the variance for the difference be-
tween two percentages would be roughly double the variance for a single
percentage based on the whole sample. Therefore, confidence intervals
would be about 40 percent larger and a difference between two domains is
probably not significant unless it is at least as large as 7 percentage points.
Interpretations of differences that are less than 7 percentage points are un-
warranted, and the IQCMP report contains many such instances.
When percentages are compared across variables, the standard error for
the difference may also be larger than the standard error for a single per-
centage based on the whole sample. However, since the statistics on each
variable are calculated from the same observations, the variables have a
nonzero covariance, and the standard error for the difference requires the
inclusion of that covariance. There is not enough information in the IQCMP
report to obtain the covariances between the variables, and so it is not
possible to say, based on the report, how large the standard error for differ-
ences across variables might be. A statement such as the following from
page II-1, "absolute student error was higher than absolute institutional
error (6.6 percent of dollars and 5.1 percent of dollars, respectively)," is
probably not warranted because the difference is probably not significant.
Similar statements appear on pages II-6 and II-15.
MARGINAL ANALYSIS
The marginal analysis was conducted by comparing an award as actu-
ally distributed with the award calculated by substituting the value from the
most reliable source for the variable under study. As discussed above,
reliability ratios would be a very useful statistic for comparing variables
and sources of information. Also, it would be much more preferable to use
an estimated true value based on a measurement-error model than to declare
one source of information to be the most reliable. The marginal analysis
found that the variables for students' (and spouses') cash, savings, and
checking, as well as various measures of income, household size, and num-
ber in college, were sources of a large amount of error.
ERROR PROFILE ANALYSIS
In addition to the marginal analyses discussed above, an error-profile
analysis was conducted to examine the joint effects of predictor variables,
OCR for page 190
190
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
such as income, assets, and year in college. Since the dependent variable in
this analysis was presence or absence of error (either student error or insti-
tutional error), a preliminary analysis was performed using chi-square tests
of association in contingency tables. All continuous variables were catego-
rized for this analysis. Note that an analysis performed with contingency
tables is still a marginal approach. The joint analysis was performed essen-
tially by multiple regression, but an unusual series of steps were taken for
the joint approach.
The preliminary analysis using chi-square tests is very close in purpose
to the marginal analyses described in an earlier section of the IQCMP re-
port, except now the dependent variable is the presence or absence of an
error rather than the magnitude of the error. Hence, one would expect to
find essentially the same results here as in the marginal analyses. It is not
clear from the IQCMP report why continuous variables were categorized.
With a categorical dependent variable and continuous independent variables,
a logistic regression, which does not require the independent variables to be
categorical, can be used for the same purpose as the contingency tables
described above. Since the logistic regression does not throw away infor-
mation by categorizing the variables, it would have been a more efficient
approach for the continuous variables than the contingency table approach
that was used. The loss of information due to categorizing the variables
would generally be no more than 5 percent, so it should not be expected that
there would be extensive differences if logistic regression had been used
instead. (Logistic regression can be performed with PROC LOGISTIC in
SAS.)
Since the categorization of continuous variables was based on percen-
tile ranking, one possible benefit of the procedure would be to reduce skew-
ness in the independent variables. If skewness was present (it is not pos-
sible to tell if it is from the IQCMP report) it could be reduced to an accept-
able level by a more straightforward transformation. With variables such as
income, the use of the log transformation effectively reduces skewness.
Although loss of information due to categorizing continuous variables is
probably not large, future studies of quality control in student financial aid
programs should use logistic regression when appropriate.
It is also not clear from the IQCMP report whether the statistical meth-
ods used for the contingency table analyses were appropriate for a complex
sample involving clustering. Standard computer packages (e.g., SAS and
SPSS) operate on the assumption of a simple random sample; packages that
are based on methods for cluster samples are widely available (e.g., Super
Carp). The erroneous use of simple random sample methods will tend to
produce p values that are too small; hence, some variables that are not related
to error in financial awards will appear to be related. Logistic regression
OCR for page 191
APPENDIX A
191
methods for cluster samples are also available (e.g., PC Carp) and would be
the more appropriate tool.
Results from the contingency table analyses showed that various mea-
sures of income, various measures of assets, household size, and marital
status were all strongly associated with student error. Among other vari-
ables, institution type and the method used by an institution to recheck data
and calculations were strongly associated with institutional error.
To assess the effects of independent variables jointly on error, the IQCMP
used a multiple regression approach. The multiple regression was based on
some unusual methodology, few details of which are given in the IQCMP
report. From the description, it appears that the dependent variable in these
regressions was the presence or absence of error, that is, still a discrete
variable. If the dependent variable is discrete, logistic regression, not ordi-
nary multiple regression, should be used. The IQCMP report should clarify
the method of estimation used here.
In preparation for the multiple regressions, continuous independent variables
were created by a transformation of dubious merit. The transformation
consisted of assigning a value to the categorical variables based on the
proportion of students with error for the relevant category from the contin-
gency table analysis. It is not clear why this transformation was used, since
the transformation does not make the variables continuous. It changes the
distance between the categories, but it does not produce a continuous vari-
able. Moreover, it is not clear why continuous variables were thought to be
necessary. For a discrete dependent variable and for discrete independent
variables, a logit analysis can be performed to accomplish the same purpose
that a multiple regression accomplishes for continuous variables. (PROC
CATMOD in SAS can be used to perform a logit analysis.)
It is also not clear whether the method of assigning a value to categori-
cal variables based on the proportion of students with error for the relevant
category from the contingency table analysis was used on only the variables
that were originally categorical or also on the variables that were originally
continuous. For variables that were originally continuous, it would be pointless
to categorize them by percentile rankings (as done for the contingency table
analysis) and then transform them back to continuous variables by using the
contingency table proportions for the multiple regression analysis. For the
multiple regressions, the variables that were originally continuous should be
used in that form, and not categorized first. The continuous variables could
be log-transformed to stabilize the variance, and/or normalized, but trans-
forming them from continuous to categorical and then back to continuous
would be difficult to justify.
The effects of cluster sampling should also be recognized in the regres-
sion analyses used for the error-profile analyses. Ordinary least squares
(OLS) regression is based on the assumption that observations are indepen
OCR for page 192
92
QUALITY IN STUDENT FINANCIAL AID PROGRAMS
dent. Since cluster sampling was used for the IQCMP, observations within
schools are correlated, and OLS should not be used to estimate parameters.
In this situation, a nested-error regression model could be used to estimate
model parameters by generalized least squares. The nested-error model has
been well developed, and the effect of using it instead of an OLS model
would be to widen confidence intervals around estimated values. That is, a
variable is less likely to appear to be significantly related to error, because
recognizing the correlation among observations within a school has the
effect of reducing the sample size. (The nested-error regression model has
been used with cluster sample data by Battese et al., 1988.) Estimation for
the nested-error regression model can be carried out with the Super Carp
computer program. Linear regression using weights to reflect the complex
sampling could also be used, but there are advantages to the variance com-
ponents approach used in the nested-error model.
Another useful approach to the joint analyses would have been to esti-
mate a covariance matrix for the (marginal) errors calculated in dollars.
This covariance matrix could have been used with exploratory multivariate
techniques, such as a principal component analysis and a canonical correla-
tion analysis. The latter analysis would have been particularly useful for
finding clusters of independent variables that relate to key dependent vari-
ables. (The Super Carp computer package can be used to calculate the
covariance matrix based on a two-stage cluster sample.)
The results of the error-profile analyses based on multiple regressions
showed that errors in reported income accounted for a large portion of the
student-based errors in financial aid awards. Whether or not a tax form was
filed, use of estimated income, and indicator variables for type of award
sought were also significant variables. Institutional variables strongly asso-
ciated with error included type of institution control, institution type, method
used to recheck files, and indicators for type of award.
SUMMARY
Some of the statistical methods used for the IQCMP were inefficient,
especially in the joint analyses for deriving error profiles. Some alternative
methodologies, such as including the reliability of variables and using esti-
mated true values, are recommended strongly for any future studies of qual-
ity control in student financial aid programs.
Despite the inefficiencies of some of the methods, there is a consis-
tency to the findings given in the IQCMP report. The marginal analyses,
the contingency table analyses, and the joint error-profile analyses all found
that reported income was an important source of error in the awards made
under the Title IV programs. Reported household size was also found to be
a source of error in many analyses.
OCR for page 193
APPENDIX A
193
The IQCMP report also addresses simplification of the financial aid
formula and potential corrective actions. It is natural to consider whether
the simplifications and corrective actions reflect the findings from the sta-
tistical analysis of errors. Several of the suggested corrective actions do
reflect findings from the statistical analysis. In particular, the suggestion
that awards not be based on tax information estimated by the applicant and
the suggestions to require specific information on household size, number
in college, and home value and debt are actions that address some of the
consistently found sources of error. However, error in reported income, the
variable most strongly associated with error over the different analyses,
may not be reduced significantly by these suggested corrective actions.
Perhaps other corrective actions should be considered for reported income.
REFERENCES
Battese, G. E., R. M. Carter, W. A. Fuller
1988 An error components model for prediction of county crop areas using survey and
satellite data. Journal of the American Statistical Association 83(401):28-36.
Cochran, W. G.
1977 Sampling Techniques. New York: John Wiley & Sons.
Price Waterhouse
1989 Sampling Plan for the Integrated Quality Control Measurement Project. Prepared
in association with Pelavin Associates, Inc. and The Gallup Organization. Wash
ington, D.C.: Price Waterhouse.
1990 Integrated Quality Control Measurement Project, Findings and Corrective Actions.
Prepared in association with Pelavin Associates, Inc. and The Gallup Organiza
tion. Washington, D.C.: Price Waterhouse.
Representative terms from entire chapter:
iqcmp report
