National Academies Press: OpenBook
« Previous: 2 Which Children Are Most Affected by Mobility?
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

3
Methodological Issues

Mobility is a complex phenomenon to study. Directly or indirectly, each of the presenters highlighted some of the methodological challenges in collecting and analyzing data that would enable them to make solid inferences about the effects of mobility on children’s development and the effectiveness of possible interventions. Mobility is not a single event that happens at a particular point in time, but a series of processes and changes that may have complex and cumulative effects— and these effects are likely to vary with the characteristics of the children who experience the mobility. Eric Hanushek and Jens Ludwig provided two perspectives on ways of identifying and assessing the impact of mobility.

ISOLATING THE EFFECTS OF MOBILITY ON INDIVIDUALS AND SCHOOLS

Hanushek began with the point that had emerged so clearly from the data already discussed—that although national residential mobility rates may have declined, they are consistently highest among low-income families. The challenge is to distinguish between positive moves, made in search of better schools and neighborhoods, and moves that are made because of some sort of disruption and that cause harm. Disentangling the effects of these two kinds of moves is difficult because researchers lack reliable measures of school quality and of family choices and behaviors, and because ways of observing the causes of mobility are limited. Never-

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

theless it is important to identify possible negative effects of mobility, both for individual students and the schools into and out of which they transition.

To address this need, Hanushek and his colleagues attempted to hold all other factors (e.g., family and neighborhood characteristics) constant and examine the independent effect of changing schools. They developed a model that uses measures of achievement growth to examine the different ways in which mobility might affect individual students and schools.1 Achievement is the product of many factors, so they made a few assumptions to simplify the analysis. First, they assumed that achievement during the school year prior to a move was not unusually bad or good. Second, they assumed that students are generally on a growth path, and that this path is likely to follow a particular trajectory, assuming a constant school quality from grade to grade. That is, if a student is learning at grade level, he or she is likely to continue at that level after moving, assuming the new school is of similar quality. Finally, they assumed that the disruption that caused the move (e.g., divorce, economic upheaval) lasted only for the year in which the move took place. These assumptions allowed them to isolate any changes in school quality that students experienced when they moved.

Hanushek and his colleagues applied the model to data from the Texas Schools Project, which provided attendance and mathematics achievement data for three cohorts of students in grades 4 through 7. For the general student population, including students who move for different reasons, they found that the move itself had little effect on the students’ academic trajectory (past the disruption in the year of the move). In other words, Hanushek said, “if they’re in a bad situation, the move has very little marginal effect on their bad situation.” More challenging was to discover whether high mobility has a measurable effect on peers, teachers, and the quality of the schools into and out of which students move. To do this, they compared student achievement among, for example, fourth graders in consecutive years, and used the differences in mobility rates “to see whether mobility shows up in differences in achievement, other things being equal.” They found that “higher student mobility in a school during the school year really hurts everybody, and it hurts people in a fairly dramatic way.”

Hanushek emphasized that for people in high-mobility schools these effects persist throughout their school careers. Moreover, African American students in the Texas sample had much higher mobility rates than other children, and they also tended to go to schools with much higher mobility. Hanushek and his colleagues estimated that the difference in mobility rates between white and African American students in Texas

1

The model is described in detail in Hanushek, Kain, and Rivkin (2004).

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

“can explain about 15 percent of the achievement gap by grade seven.” Participants questioned the representativeness of the Texas study, and particularly the assumption in that study that the effects of mobility are temporary.

Hanushek concluded that understanding the impact of mobility is difficult because measuring the characteristics of schools and families that make a difference is so difficult. But, by isolating individual fixed effects, they were able to show that while the effects of mobility on individuals are fairly small, the effect on schools is quite large.

SEEKING CAUSALITY

Jens Ludwig described the methodological challenges associated with estimates of the effects of residential mobility on outcomes for children. As others had already discussed, there are numerous ways in which mobility might affect development, positively or negatively. Moving may be disruptive, and it also changes the social networks, routines, and perhaps the available coping strategies in a child’s life. After a move, a child may have more, fewer, or simply different community, social, institutional, and physical resources. Because the reasons for a move and its consequences are so varied, it is difficult to think about a single effect of moving. The implications for child development and schooling are likely to depend in significant ways on the motivation, circumstances, and options facing a family that moves.

Thus, to understand the effect of a move, it is necessary to understand the circumstances the family might have experienced if they had not moved—or the “counterfactual” condition. For example, if the head of a household loses his or her job, the family is evicted from their home, and they move in with relatives, should the counterfactual be identified as the scenario in which the job was not lost, the job was lost but the family had the wherewithal to avoid eviction, or the family was evicted but moved into a different sort of neighborhood? Alternatively, if a head of household receives a promotion and a raise in pay and moves the family to a wealthier neighborhood, is the counterfactual the adult is not getting the promotion, the family not moving, or the family not moving but sending their children to private school?

In seeking the correct counterfactual, one might consider the various constraints on the choices families are making. A family that experiences a loss of income may or may not move, but it will have to adjust in some way, and most of the possible adjustments could affect children’s development. So, Ludwig suggested, from a social science point of view, one important implication is that in conducting statistical analyses it is possible to “overcontrol” as well as to “undercontrol” for relevant confounding factors. That is, some of the explanatory variables researchers

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

include in their models may be factors that families would be forced (or able) to adjust whether or not they chose to make a move. The most promising way to understand the messy world of people’s choices can best be studied in terms of questions about how they respond to particular interventions, whether they are policy interventions or natural experiments that are induced by changes in the economy or the housing market. If one begins with this sort of policy evaluation question, it can be researched using a variety of observational approaches, in which two groups that naturally experience different treatments (without intervention) are compared and the researcher controls for factors other than the treatment that might have affected the outcome. Alternatively, one might use a randomized approach, in which subjects are randomly assigned to different treatment groups, or so-called natural or quasi-experiments, in which the treatment is “assigned” by a change of policy or other factor beyond the control of subjects.

These distinctions have stimulated passionate disagreements about establishing causation in many contexts, and, Ludwig noted, an empirical literature has emerged that describes ways of using nonexperimental estimation when a randomized study is not feasible. For example, Robert LaLonde (1986) developed an influential approach to evaluating estimation methods. Researchers who have data from a randomized experiment, and therefore know the “right” answer regarding the effect of a policy intervention on particular outcomes, can then try to use different observational methods to see if it is possible to reproduce the results that were obtained experimentally. “The answer that LaLonde got,” Ludwig explained, was that there were big differences between the two sets of results, a finding that “was shockingly grim and has had a profound influence on the field of empirical economics and applied statistics.”2

This approach was initially developed for research on employment, but it has been used to examine education questions as well. Although the results of nonexperimental estimates vary in practice, depending on the context and the quality of the available data, Ludwig has found that, on average, they do not look particularly impressive. He described a comparison between experimental and nonexperimental results related to mobility to illustrate the problem, using 1990s data from the U.S. Department of Housing and Urban Development’s (HUD’s) Moving to Opportunity (MTO) study. In that study, a total of 4,600 families in 5 cities were recruited to participate and randomly assigned to 1 of 3 groups. Two of the groups received different sorts of housing vouchers designed to help them move to neighborhoods with lower poverty rates, and the

2

Other researchers have done similar work with somewhat different results (see Dehejia, 1999; Heckman, 2008; Smith and Todd, 2005).

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

control group received no voucher. Thus, Ludwig explained, “the random assignment resulted in nontrivial differences in the average neighborhood environment for otherwise similar types of families.” These results could be used to compare the effects of different specific neighborhood characteristics and mobility on outcomes for children.

To develop a nonexperimental estimate for comparison with the empirical results of the randomized experiments, researchers can use different methods, such as standard regression analysis and its close cousin, propensity score matching, in which certain variables are held constant and others are varied with the goal of isolating a particular effect. The ability of these sorts of nonexperimental or observational approaches to reproduce the experimental answer probably depends on the quality of the data that are available, so it is important that a fairly rich set of background characteristics are available for the families and children in the MTO study, including demographics (e.g., age, household size), socioeconomic data (e.g., income, parental employment, history of public assistance), housing and mobility history, perceptions of the baseline neighborhood (e.g., having local family or friends, perception of safety), and, perhaps most important, the family’s motivation for wanting to move and past experiences in different types of neighborhoods.

Figure 3-1 shows the results of the comparison between the empirical results of the randomized experiment and the estimated results of applying nonexperimental methods to the MTO data on the effects of housing vouchers. The light gray bars, labeled OLS, are the estimated, nonexperimental results for various outcomes, and the dark gray bars, labeled IV, are the experimental results. The estimated results indicate, for example, that boys who live in high-poverty areas are slightly less likely to be involved in risky behavior than boys who live in low-poverty areas. However, the darker bar indicates that the experimental results were quite different. This result occurs, Ludwig suggested, because factors not observed in the study lower the likelihood that these boys would engage in risky behaviors. In his view, the stark disparity on the outcomes for the two different approaches to examining mobility in this context makes clear that the nonexperimental method rested on assumptions that the empirical evidence indicates were wrong. However, he pointed out, randomized or natural experiments can reduce the selection bias that limits the usefulness of the estimates, but they also limit the range of questions that can be explored.

Ludwig argued that the research portfolio regarding the factors that determine mobility is currently out of balance. He suggested that randomized and natural experiments should make up a larger proportion of the research portfolio than they do now. At the same time, rich descriptive

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
FIGURE 3-1 Comparison between empirical and estimated results.

FIGURE 3-1 Comparison between empirical and estimated results.

NOTE: OLS = the estimated, nonexperimental results for various outcomes; IV = the experimental results.

SOURCE: Adapted from Kling, Liebman, and Katz (2007).

studies would support the design of better experiments down the road, by highlighting possible mechanisms and other researchable questions.

Participants generally liked his approach, although several reiterated the point that using strict randomly controlled trials to study families’ housing decisions and other questions related to mobility is often not feasible and perhaps unethical. Moreover, recent improvements in nonexperimental methods mean that more feasible alternatives exist.3 Several stressed the value of natural experiments, looking at, for example, the impacts of foreclosures on students’ academic achievement. Another suggested that, given the practical and ethical difficulties of randomly controlled studies, the solution is to seek “overwhelming data of the

3

Nonexperimental research designs that might be used when randomized controlled studies are not feasible include more qualitative approaches, such as case studies or ethnographic surveys; longitudinal studies; correlational studies; and statistical analyses, such as regression discontinuity, use of instrumental variables, or propensity score matching.

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

weaker kind that [describe how] a process works in the natural world.” These discussions suggest that a combination of rich qualitative reports, studies using the weaker regression design, and a theoretical model about the mechanisms through which an intervention works might be very compelling.

Ludwig acknowledged the point, adding that because so many families who would be eligible for housing assistance are not receiving it, there are multiple opportunities for comparing outcomes among different groups. Ludwig stressed that answering questions about causation is most important at the point when policy makers are contemplating a specific intervention. The key concern with nonexperimental methods, he suggested, is that they can give a misleading reading on the size or even the existence of key causal relationships. More important, he said, “We currently do not have very good methods for determining how well our nonexperimental approaches are working. If we cannot be sure that our nonexperimental estimates are close to the truth we must be appropriately cautious in basing policy decisions on such results, and continue to push harder to develop a body of evidence that gives us greater confidence that we understand what the consequences of different policy decisions will be.”

Reflecting on the state of mobility research in light of the methodological issues that had been raised, Stephen Raudenbush offered several observations. First, he noted that the field has benefited recently from a body of carefully done empirical studies that represent an important step forward. These studies have established a strong basis for conclusions about which children are exposed to the most mobility and which children seem to be at greatest risk for harm from mobility. But the causal questions are complex, and the interpretations of what these data mean are less straightforward.

Raudenbush highlighted the value of Hanushek, Kain, and Rivkin’s approach and Ludwig’s challenge. Their model bypassed the lack of complete data to show the short-term disruptive effects of a move in a way that takes into account the effects of the new school and neighborhood of a move. It also has a useful way to reframe the causal question by looking for the impact of attending a school that is characterized by high levels of mobility. This is valuable, Raudenbush observed, because it demonstrates the need for a policy intervention based on the results of this causal question. For his part, Ludwig challenged researchers and policy makers not to take shortcuts on causal questions. Strong design considerations—either randomized experiments, regression discontinuity studies in which the selection process is fully known, or natural experiments in which there is a clear instrumental variable—are needed to support causal interpretations.

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×

This page intentionally left blank.

Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 33
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 34
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 35
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 36
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 37
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 38
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 39
Suggested Citation:"3 Methodological Issues." National Research Council and Institute of Medicine. 2010. Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop. Washington, DC: The National Academies Press. doi: 10.17226/12853.
×
Page 40
Next: 4 Policy and Programmatic Responses »
Student Mobility: Exploring the Impacts of Frequent Moves on Achievement: Summary of a Workshop Get This Book
×
Buy Paperback | $29.00 Buy Ebook | $23.99
MyNAP members save 10% online.
Login or Register to save!
Download Free PDF

Many low-income families struggle with stable housing and frequently have to move due to foreclosures, rent increases, or other financial setbacks. Children in these families can experience lasting negative effects, especially those who are young and still developing basic learning and social skills. A joint NRC-IOM committee held a workshop in June 2009 to examine these issues, highlight patterns in current research, and discuss how to develop a support system for at-risk children.

  1. ×

    Welcome to OpenBook!

    You're looking at OpenBook, NAP.edu's online reading room since 1999. Based on feedback from you, our users, we've made some improvements that make it easier than ever to read thousands of publications on our website.

    Do you want to take a quick tour of the OpenBook's features?

    No Thanks Take a Tour »
  2. ×

    Show this book's table of contents, where you can jump to any chapter by name.

    « Back Next »
  3. ×

    ...or use these buttons to go back to the previous chapter or skip to the next one.

    « Back Next »
  4. ×

    Jump up to the previous page or down to the next one. Also, you can type in a page number and press Enter to go directly to that page in the book.

    « Back Next »
  5. ×

    Switch between the Original Pages, where you can read the report as it appeared in print, and Text Pages for the web version, where you can highlight and search the text.

    « Back Next »
  6. ×

    To search the entire text of this book, type in your search term here and press Enter.

    « Back Next »
  7. ×

    Share a link to this book page on your preferred social network or via email.

    « Back Next »
  8. ×

    View our suggested citation for this chapter.

    « Back Next »
  9. ×

    Ready to take your reading offline? Click here to buy this book in print or download it as a free PDF, if available.

    « Back Next »
Stay Connected!