This negative association between age and outcome in the United States is even more evident when we control for grade, though we do not show that effect here. The U.S. scatterplot excludes a small number of cases with extreme ages.

We evaluated an alternative hypothesis that nonnative speakers of English may be overrepresented among the older Population 2 students in the United States. About 12.4 percent of the U.S. sample did speak a language other than English at home, while no comparable variable exists in the Japanese data set. Yet language at home was not related to age in the U.S. data. Thus, although language at home is likely a confounding variable (it is plausible that the U.S. sample has more nonnative English speakers than the Japanese sample has nonnative speakers of Japanese), this does not explain the negative association between age and achievement in the United States. We are left with grade retention as the more plausible explanation. Other explanations involving differential selection of students into the U.S. and Japanese samples cannot, however, be ruled out.

By defining the population in terms of grade, the design effectively conditions inferences about countries on an endogenous variable rather than the desired exogenous variable.

The presence of students who have been retained in seventh and eighth grades contradicts the notion that Population 2 is a “cohort” (where cohort is defined as a subpopulation born during a given interval of time). Similarly, the absence of sixth graders who were retained also selects cases from a well-defined cohort. Thus a comparison between Populations 2 and 1 is not really a comparison of cohorts, and the meaning of the comparison is ambiguous.

This benefit is not likely, however, to extend to comparisons of Population 3. Dropout rates will likely vary over time and the notion of “school leaving” appears to be changing as well; for example, consider the changing role of community colleges in the United States.

An analysis using this type of model appears in Gamoran (1991), who used data from the Second International Mathematics and Science Study (SIMS) to assess curriculum effects. A major advantage of this analysis is that SIMS provided longitudinal data at the student level, thus allowing control of prior achievement. TIMSS does not.