Next Steps for TIMSS Directions for Secondary Analysis (1999) / Chapter Skim
Currently Skimming:
Recommendations for Making the Most of TIMSS
Recommendations for Making the Most of TIMSS
Pages 11-27
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 11... ...
Moreover, while the reports that are part of the primary work of TIMSS were all reviewed, they are nevertheless official reports that have not received the kind of open peer review to which independent scholarship is generally subject. The board urges that each of the scholarly communities with an interest in TIMSS explore the hypotheses suggested by TIMSS, the data that have been collected, and the methodological issues the study has raised.
|
From page 12... ...
As David Baker noted, it is the cross-national design that will "enable TIMSS to have its biggest impact on policy formation in the United States," (Baker, 1998:5) , because this is what makes it possible to identify system-level factors that vary across countries such as school governance, standards enforcement, textbook dominance, and many other features and look at how they relate to differences in national performance.
|
From page 13... ...
Richard Murnane made a similar point when he stressed the importance of comparing across nations the degree of variation in student achievement within nations, and tying this variation to measures of socioeconomic inequality (Murnane, 1998~. Similarly, Elizabeth King advocated such comparisons of within-country variation because of what they can reveal about "the importance each country accords equality across groups or its success in achieving this equality" (King, 1998:3~.6 By going deeper than the achievement rankings in this way, they and others argued, researchers can make much more useful connections between TIMSS and the kinds of policy questions that are of most interest in the United States Mary Metz and numerous others remarked on the value of cutting the data in different ways than has been done and comparing, within countries, subgroups of schools, classes, and students; subsets of the domain; performance on particular item types; etc.
|
From page 14... ...
For example, the selection of questionnaire topics reflects a variety of assumptions about the role of school structure and many other "inputs" in student learning; the design of the curriculum study reflects some assumptions about the relationship between teaching and curricular exposure; decisions made in the course of video and case studies reflect implicit and explicit ideas about factors that are most likely to influence learning and how those might be explored. It is important now to look back and assess the extent to which the results fit the study model.
|
From page 15... ...
For example, participants with qualitative backgrounds were disinclined to think that the questionnaire data regarding professional development for teachers was of much use, while more quantitatively inclined participants were equally cautious about how the videotapes and the case studies could best be used. The conversations that pushed participants with different views to consider new perspectives made plain that simple resolutions for many of the issues are not likely.
|
From page 16... ...
The board believes it will be integral not only to making the most of TIMSS, but also to moving comparative education research forward and enabling it to provide information that can be of real value for decision makers. Recommendation: Use the Problems and Successes of TIMSS to Inform the Planning of Future Studies Testing and exploration of the conceptual framework of TIMSS is an important step in the long-term intellectual inquiry of which TIMSS is a part.
|
From page 17... ...
RECOMMENDATION: EVALUATE CLAIMS THAT HAVE ALREADY BEEN MADE BASED ON THE DATA Role of Curriculum One particular set of claims based on TIMSS data that crossnational variation in academic achievement is influenced by and accounted for by cross-national variation in the content of curricular frameworks for science and mathematics, and that the U.S. students' weak performance can be attributed to a curriculum that is "a mile wide and an inch deep" was the subject of special attention at the workshop.8 BICSE members chose to address this claim in part because it is the most specific causal claim that has been put forth in TIMSS reports and in part because this claim has received a significant amount of public attention.
|
From page 18... ...
curricula and within the network of assessments, standards, professional development for teachers, and other factors that affect learning that sets the United States apart from other countries and actually accounts for the weaknesses U.S. students demonstrated on TIMSS received even greater challenges.
|
From page 19... ...
Moreover, as Gary Natriello suggested, analyses that leave citizenship aside and group the sampled population according to characteristics of their educational experience might be very useful in exploring the ways the intended and implemented curricula interact and affect learning (Natriello, 1998:5~. A number of participants shared the view that the reports of the curriculum study may not have distinguished adequately between exposure to curriculum and the instructional modes through which the curriculum was delivered.
|
From page 20... ...
Follow-up research is needed to pursue the most intriguing implications of and questions about this data and to provide policy makers and practitioners with a more nuanced picture of the role of curriculum in student learning. Other claims that have been, or may be, made based on the TIMSS data require follow-up as well.
|
From page 21... ...
Cognizant of these criticisms, the TIMSS designers addressed the issue early in the process. Having learned from SIMS that it would not be possible to ensure complete compliance with all sampling procedures in all countries, they made the existence of sampling compromises in particular countries very clear in the published reports and in the released data.
|
From page 22... ...
As is discussed elsewhere in this report, more detailed portraits of the education systems and contexts for learning in other countries could be of great value as researchers attempt to explore puzzles in the data and pursue specific comparisons between individual countries. Germany and Japan are obvious targets for future research because of the threecountry studies that were part of TIMSS, but high-achieving countries such as Singapore, and others that stand out for various reasons, may be producing research that follows up on their own national data.
|
From page 23... ...
Without in any way disparaging the results already produced by the TIMSS Study Center, the board strongly recommends that alternative models for creating achievement scores be explored by independent scholars. Given that the achievement results, and the rankings in particular, have been accorded a significant measure of political importance within the United States, it is vital that any additional insights these data might yield be mined and made public.
|
From page 24... ...
For example, Senta Raizen noted that "there simply isn't enough information available in any of the TIMSS datasets to differentiate sufficiently among countries either as to training and support received by teachers in each country or regarding teachers' willingness to teach in a particular way, let alone link these variables" (Raizen, 1998:7~.
|
From page 25... ...
Many of the TIMSS researchers have assisted the board in its efforts to keep track of the data that have been collected and to understand the structures of the various datasets. The board greatly appreciates this assistance and has taken note of the efforts that have been made by the TIMSS Study Center staff, the National Center for Education Statistics, and others to release the data as quickly as possible and to make datatapes, users' guides, and the like available on the web and elsewhere; to hold training sessions; and to field questions.
|
From page 26... ...
Workshop discussions about possibilities for linking one part of the study to another made clear that it is not efficient for independent scholars to, for example, spend time tracking down correspondence among different sets of data; this is work that, once done, should be made easily available to anyone considering possibilities for secondary analysis. It is also clear that both relatively straightforward issues of translation, as well as more potentially significant questions about cultural context the precise meaning of the words in different languages for "homework," for example could be a hindrance to many scholars.
|
From page 27... ...
RECOMMENDATION: INDEPENDENT SCHOLARS OR TEAMS OF SCHOLARS SHOULD RECEIVE FUNDING TO CONDUCT SECONDARY ANALYSIS As this report has attempted to make clear, the board would like to see a broad array of scholars from a variety of disciplines use the TIMSS data in a variety of ways. Much of the funding for such work will need to come from the institutions that have already funded much of TIMSS and from others with significant resources for education research.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.