. "9. Statistical Issues in Analysis of International Comparisons of Educational Achievement." Methodological Advances in Cross-National Surveys of Educational Achievement. Washington, DC: The National Academies Press, 2002.
The following HTML text is provided to enhance online
readability. Many aspects of typography translate only awkwardly to HTML.
Please use the page image
as the authoritative form to ensure accuracy.
Methodological Advances in Cross-National Surveys of Educational Achievement
This negative association between age and outcome in the United States is even more evident when we control for grade, though we do not show that effect here. The U.S. scatterplot excludes a small number of cases with extreme ages.
We evaluated an alternative hypothesis that nonnative speakers of English may be overrepresented among the older Population 2 students in the United States. About 12.4 percent of the U.S. sample did speak a language other than English at home, while no comparable variable exists in the Japanese data set. Yet language at home was not related to age in the U.S. data. Thus, although language at home is likely a confounding variable (it is plausible that the U.S. sample has more nonnative English speakers than the Japanese sample has nonnative speakers of Japanese), this does not explain the negative association between age and achievement in the United States. We are left with grade retention as the more plausible explanation. Other explanations involving differential selection of students into the U.S. and Japanese samples cannot, however, be ruled out.
By defining the population in terms of grade, the design effectively conditions inferences about countries on an endogenous variable rather than the desired exogenous variable.
The presence of students who have been retained in seventh and eighth grades contradicts the notion that Population 2 is a “cohort” (where cohort is defined as a subpopulation born during a given interval of time). Similarly, the absence of sixth graders who were retained also selects cases from a well-defined cohort. Thus a comparison between Populations 2 and 1 is not really a comparison of cohorts, and the meaning of the comparison is ambiguous.
This benefit is not likely, however, to extend to comparisons of Population 3. Dropout rates will likely vary over time and the notion of “school leaving” appears to be changing as well; for example, consider the changing role of community colleges in the United States.
An analysis using this type of model appears in Gamoran (1991), who used data from the Second International Mathematics and Science Study (SIMS) to assess curriculum effects. A major advantage of this analysis is that SIMS provided longitudinal data at the student level, thus allowing control of prior achievement. TIMSS does not.
Blumberg, C., & Porter, A. (1983). Analyzing quasi-experiments: Some implications of assuming continuous growth models. Journal of Experimental Education, 51, 150-159.
Bryk, A., & Weisberg, H. (1977). Use of the non-equivalent control group design when subjects are growing. Psychological Bulletin, 84, 950-962.
Campbell, D., & Erlebacher, A. (1970). How regression artifacts in quasi-experimental evaluations can mistakenly make compensatory education look harmful. In J. Hellmuth (Ed.), Compensatory education: A national debate, volume 3: The disadvantaged child (pp. 185-210). New York: Brunner/Mazel.
Gamoran, A. (1991). Schooling and achievement: Additive versus interactive models. In S. W. Raudenbush & J. D. Willms (Eds.), Schools, pupils, and classrooms: International studies of schooling from a multilevel perspective (pp. 37-52). San Diego, CA: Academic Press.
Holland, P. (1986). Statistics and causal inference. Journal of the American Statistical Association, 81(396), 945-960.
Little, R., & Rubin, D. (1987). Statistical analysis with missing data. New York: John Wiley & Sons.