|
Items in cart [4] |
Questions? Call 888-624-8373 |
PAPERBACK Free Resources |
||||||||||||||||
|
||||||||||||||||
|
|
|||||||||||||||
| ||||||||||||||||||||||||||||||
|
||||||||||||||||||||||||||||||
| Copyright © 2009. National Academy of Sciences. All rights reserved. Terms of Use and Privacy Statement |
Below are the first 10 and last 10 pages of uncorrected machine-read text (when available) of this chapter, followed by the top 30 algorithmically extracted key phrases from the chapter as a whole.
Intended to provide our own search engines and external engines with highly rich, chapter-representative searchable text on the opening pages of each chapter.
Because it is UNCORRECTED material, please consider the following text as a useful but insufficient proxy for the authoritative book pages.
Do not use for reproduction, copying, pasting, or reading; exclusively for search engines.
OCR for page 3
of methodological questions to intense consideration of the structure
of public education in the United States. As the symposium planners
intended, the presentations and discussion focused not on achieving
group consensus, but on unearthing a variety of views on a complex
topic. (See Appendix C for a list of the papers presented.) Clearly
the day and a half allotted for the symposium did not allow for an
exhaustive discussion of either the strengths and weaknesses of TIMSS
or its many implications for policy makers. Moreover, because of
time constraints, a number of important points were raised but not
elaborated during the discussion.
This summary report is an additional component of the effort to
foster dialogue in the education research and policy communities. It
describes the major elements of TIMSS, presents some of the discus-
sion that took place at the symposium, and explores the themes that
emerged from it. Because TIMSS is so complex, the steering com-
mittee charged with planning the symposium decided to devote con-
siderable symposium time to explication of the structure of the study
and a few of its principal findings. This document follows that lead.
The next section, "What Is TIMSS?," provides a description of the
study and of the presentations made by the TIMSS researchers. The
following two sections summarize, respectively, the questions and
critiques that presenters raised about the study itself and the major
policy issues that were addressed. The last section summarizes the
major ideas that emerged at the symposium.
WHAT IS TIMSS?
As its name indicates, TIMSS is the third in a series of investiga-
tions of mathematics and science learning conducted under the aus-
pices of the International Association for the Evaluation of Educa-
tional Achievement. IEA is an international consortium of research
institutions in more than 40 countries. Although individual govern-
ments may fund their countries' participation in IEA activities, the
organization is run by an assembly of country representatives. The
first IEA study, of mathematics, was conducted in the 1960s; the
second mathematics study was done in the 1970s. IEA has also
conducted studies of learning in a variety of other subjects. Although
the structure and composition of IEA's studies have evolved some
since the 1960s, their purpose to describe and explain differences in
student achievement has remained the same.
More specifically, the organizers of the study described the pur-
pose of TIMSS in this way: "to learn more about mathematics and
science curricula and teaching practices associated with high levels
of student achievement, in order to improve the teaching and the
learning of mathematics around the world" (Robitaille and Garden,
1996:15). Study planners recognized that to accomplish this goal
they would need to collect a variety of different kinds of data. First,
they needed the kind of common measure of achievement used in
previous studies numbers that would represent the varying degrees
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
3
OCR for page 4
to which students around the world have learned the body of math-
ematics and science knowledge deemed (through international con-
sensus) essential. This was obtained by means of an achievement test
(described in greater detail below). All of the other components of
TIMSS were designed to provide data that can help explain variations
in performance on the achievement test: these included a detailed
look at the content of mathematics and science curricula and text
books around the world, as well as investigations of student attitudes
and experiences, teaching practices and school resources, and many
other factors that affect achievement (these other components of the
study are described below). The challenge for TIMSS researchers,
and for others wishing to use the data for additional analyses, is to
make full use of this combination of information about the education
practices and contexts that influence student learning.
The scope of TIMSS is unprecedented in several ways. Though
many international comparative assessments have been conducted, none
has assessed student learning in two subjects in so many countries at
the same time. Those involved in the planning and design of the
study paid considerable attention to the experience gained in the study's
predecessor, the Second International Mathematics Study (SIMS)
(McKnight et al., 1987; Medrich and Griffith, 1992~. They addressed
many of the criticisms leveled at SIMS, both by adhering to strict
sampling procedures and by expanding the scope of the design for
TIMSS to include the collection of an extensive variety of contextual
data (Rotberg, 1990; Bracey, 1996; Third International Mathematics
and Science Study, 1996~. In addition, the designers of TIMSS incor-
porated research methods from several different disciplines in a ground-
breaking effort to link different kinds of data. Essentially, several
distinct studies were conducted, each investigating questions about
mathematics and science learning from a different perspective. The
combination of different research methods raised a variety of issues
and questions, some of which are addressed below ("Critiques and
Mathodological Issues". (See Appendix D for a bibliography of TIMSS
reports and resources.)
The different components of the study grew out of three basic
questions that it was designed to answer: What are students in each
nation expected to learn? What, and how, are students actually taught?
What do students actually learn? TIMSS researchers used the terms
"intended, implemented, and achieved curricula," respectively, to re-
fer to these three basic questions (Robitaille and Garden, 1996~.
The Achievement Study
The core of TIMSS is an assessment of student achievement in
mathematics and science, administered to students at ages 9 (Popula-
tion 1), 13 (Population 2), and 17 (Population 3~. The achievement
results, of course, provide the data on the achieved curriculum what
students have actually learned. The content to be tested in each sub
4
LEARNING FROM TIMSS:
OCR for page 5
ject and at each age level was determined through a sometimes con-
tentious consensus process involving all of the participating coun-
tries. The resulting framework document, which guided the develop-
ment of the test questions, reflects many compromises; it does not
reflect the actual curriculum in any one country, and each country is
free to conduct further analyses on just those questions that covered
the material taught to its own students (International Association for
the Evaluation of Educational Achievement, 1996a, 1996b).
The test itself is similar to other large-scale assessments that are
used in the United States, such as the National Assessment of Educa-
tional Progress (NAEP). It is a combination of multiple-choice ques-
tions and open-ended exercises that ask students to generate solutions
to problems or to answer questions in their own words. The open-
ended exercises are scored using guidelines that describe several cat-
egories of responses and assign scores to them. In each country the
test was administered to a sample of classes of students approxi-
mately 3,750 students per country at each grade level (Third Interna-
tional Mathematics and Science Study, 1996~. The samples were
chosen so that various groups were adequately represented and each
country's overall population characteristics were reflected. Each stu-
dent answered only a portion of the questions meant for his or her
grade level; various subsets of the questions were printed in different
test booklets so that an appropriate number of students in each sample
would take each possible combination of questions. Consequently,
data could be reported on the entire content domain covered by the
test although each student sat for only 60 or 90 minutes of testing.
The complex item sampling design made it possible for researchers
to report on the performance of different population groups and on
student performance for different types of questions and different
content areas. The sampling procedure also made possible the so-
called "horse race" results, which rank the performances of partici-
pating countries. Results are being reported for nations and, in the
United States, for three states and one consortium of school dis-
tricts.~ Forty-one countries participated in the assessment of middle-
school, or Population 2, students (13-year-olds); these results were
released shortly before the symposium. Twenty-six nations partici-
pated in the elementary school, or Population 1, portion (9-year-olds),
results for which were released in June 1997. Data for Population 3,
students at the end of secondary school (17-year-olds), are scheduled
for release in February 1998.2 No individual scores are available.
iThe three states, Colorado, Illinois, and Minnesota, and the First in the World
Schools, a consortium in the northwest suburbs of Chicago, provided funds for their
participation as "mini-nations" in order to learn how their own students compare to
others internationally. NCES has made it possible for other states or districts who
wish to administer TIMSS locally to do so.
2Symposium participants repeatedly stressed the importance of recognizing, when
drawing interpretations from TIMSS, that different groups of nations participated in
different portions of the project. See note 5 on page 17 for the numbers of countries
participating in each major component.
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
s
OCR for page 6
Because the results are based on the performance of representative
samples of students in each country, they actually, as TIMSS researchers
explained, "represent a range within which the nation's actual average
would most likely fall if all students were tested" (TIMMS U.S. Na-
tional Research Center, 1996~. Thus, the U.S. achievement results
were presented in terms of three bands groups of countries that per-
formed better than the United States did, at approximately the same
level as the United States, or worse than the United States. By pre-
senting the results this way, researchers hoped to discourage observ-
ers from focusing on slight differences that might be inappropriately
magnified if numerical scores were simply listed in rank order.
More than 20 countries also chose to include a set of performance
assessment tasks for Populations 1 and 2; these were simple experi-
ments using standardized materials provided in kits. The tasks were
too expensive and time-consuming to include for the entire testing
population, but they are expected to yield data on skills not easily
measured by paper-and-pencil assessments (National Center for Edu-
cation Statistics, 1996~. Testing of Population 3 students also ad-
dressed two "specialist" subpopulations: students enrolled in advanced
mathematics or physics courses.
Background Questionnaires
At the time the assessments were administered, students, teachers,
and school officials were also asked to fill out background question-
naires designed to elicit important information about the contexts in
which student learning occurs. These questionnaires collected data on
students' and teachers' backgrounds, school structures and resources,
students' and teachers' attitudes about mathematics and science, teachers'
pedagogical beliefs and practices, classroom coverage of various math-
ematics and science topics, and other variables. Responses to these
questions can then be correlated with achievement data to reveal asso-
ciations between various factors and student performance. Although
such associations cannot support specific causal inferences, they can
call attention to factors that are associated with success and identify
promising areas for further study.
Quality Control
The planners for TIMSS took great care to ensure the quality of
the data collection, and independent observer Edward Haertel com-
mented on the high quality of the sampling and data collection in the
paper he presented at the symposium. The research team paid particu-
lar attention to the sampling in part because SIMS, its predecessor,
was criticized for using sampling methods that may have distorted the
international comparisons. An entire volume documenting the quality
control procedures used in TIMSS has been published (Third Interna-
tional Mathematics and Science Study, 1996), but it is worth noting
one strategy in particular. Because the sampling rules were so rigor
6
LEARNING FROM TIMSS:
OCR for page 7
ous and complicated, not all countries were able to meet all of them,
but the data collected from these countries were still of value. The
TIMSS research team defined several levels of compliance, which
were clearly indicated in the main ranking tables. Thus, readers
could see easily that comparisons between nations with differing lev-
els of compliance should be made with caution and with an under-
standing of the nature of these differing levels.
Albert Beaton, TIMSS study director, presented a brief summary
of the study at the symposium and highlighted a few of the key
results from the Population 2 data, the first data to be released and
the only data available at the time of the symposium.3 He began by
noting that, while there has been worldwide interest in the country
rankings, members of the press had not really addressed the more
complex findings of the study or the issues and the questions they
raise. For example, Beaton showed a table depicting results for the
41 Population 2 countries, similar to those used in the published
reports. He explained that a reporter from a national news magazine
had declined to publish it on the grounds that it was too complicated.
Perhaps the most striking finding for Beaton was that all of the re-
porting countries show a connection between socioeconomic factors
and performance. In every one of the 41 countries, he explained,
"there is a relationship between the number of books in the home and
school performance." There was a similarly clear relationship across
countries tested between parents' levels of education and student per-
formance. Other factors explored in TIMSS did not demonstrate
such clear relationships:
for example, class size shows some rela-
tionship to achievement, except that Korea, whose performance was
second only to Singapore's, averages more than 40 students per class.
Beaton presented some other key findings:
· There are differences in performance on particular content
areas covered by the assessment that are consistent with differences
in curricula across countries.
.
U.S. seventh-grade students ranked higher among nations than
did U.S. eighth-grade students. Beaton remarked that this finding is
important because it supports the overall achievement differences that
were found. That is, differences between grades within a nation
cannot be explained away by a large national difference, which would
have affected performance at both grades equally.
· Within most countries and overall, boys had significantly higher
mean science achievement than did girls in both the seventh and
eighth grades. Gender differences in mathematics achievement were
small or nonexistent; differences that did exist favored boys.
There is a large difference in average science and mathemat-
~cs achievement between the top-performing and bottom-performing
.
.
3Population 2 covered the two school grades containing the largest numbers of 13-
year-olds, grades seven and eight in the United States.
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
7
OCR for page 8
countries. Despite this large difference, when countries were ordered
by average achievement, there were only small differences in achieve-
ment between each country and the ones closest to it.
· In science, students generally had the most difficulty with the
chemistry items.
· In mathematics, the questions that stood out as most difficult
called for multistep problem solving and applications.
· In both mathematics and science, country performance in dif-
ferent content areas seemed to correspond to curricular emphasis.
Beaton was the first of many at the symposium to point out that
the TIMSS data has, not surprisingly, failed to produce a "silver bul-
let" that will magically transform mathematics and science education.
As Beaton put it: "Wouldn't it be nice to just find that all we have to
do is something simple, you know, increase the school year, for ex-
ample? . . . We have been poring over the data . . . and there is just
no simple answer." For every likely looking connection between achieve-
ment and a variable such as amount of homework or class size, TIMSS
showed counterexamples. Beaton and his colleagues concluded that,
while each probably has an effect, none by itself made a major differ
ence.
The Curriculum Study
As even casual observation reveals, there are substantial differ-
ences among the education systems and curricula in use in the partici-
pating nations. The purpose of the curriculum study was to find a
way to make sense of these differences and to make it possible to
explore the relationship between curriculum and achievement results.
More specifically, researchers hoped that by looking systematically at
which topics are covered at which levels around the world, and at
performance expectations, they could gain understanding of differ-
ences in student performance on particular skills and segments of the
content that were tested. This study, of course, primarily explored
what study planners called the intended curriculum.
Undertaking a thorough comparison among the curricula of 46
countries was complicated by the fact that there is no common way of
even describing curricula. The solution to this problem was a proce-
dure called topic trace mapping, by which researchers in each country
collected information about topic coverage in various documents and
translated it into a common format. Using formally defined "docu-
ment analysis procedures" as guides, the national researchers took the
most widely used textbooks in their respective countries, as well as
national and regional curriculum guides, and analyzed the documents
section by section to determine the extent to which material included
in the TIMSS frameworks was covered. A total of 491 curriculum
guides and 638 textbooks were analyzed. The researchers also asked
education experts within each country to respond to questionnaires
8
LEARNING FROM TIMSS:
OCR for page 9
designed to support the document analyses (Schmidt et al., 1997;
TIMSS U.S. National Research Center, 1996~.
William Schmidt, who directed the curriculum study, described
some of the team's findings and conclusions, focusing primarily on
the issues addressed in A Splintered Vision, the curriculum analysis
results for the United States (Schmidt, McKnight, and Raizen, 1997~.4
For him, the study's most valuable product is what he sees as resolu-
tion of the debate over whether school curricula truly make a differ-
ence in student learning. For him it is clear that teaching matters,
and he argues that the "somewhat disappointing" achievement results
for the United States reflect the weaknesses in the U.S. curricula.
His conclusion is that many other factors such as length of time
spent in school and assignment of homework that have been blamed
for poor student performance in the United States are side issues. He
explained that his research has shown that "there is a tremendous
amount of variability across these countries in terms of the way in
which mathematics or science is taught." He suggested that further
exploration of the relationship between achievement and topic cover-
age in the curriculum will clarify the picture of student learning con-
siderably.
Specifically, Schmidt argued that no intellectually coherent vi-
sion guides mathematics and science curriculum development in the
United States. Because responsibility for curriculum decisions rests
with states and localities, there is variation among the curricula used
within U.S. borders, just as there is among those of different nations.
Some of this variation reflects differing educational goals and phi-
losophies, while some of it is, in effect, coincidental. Schmidt pre-
sented a few specific findings to illustrate his points:
.
.
Both science and mathematics textbooks in the United States
include far more topics than was typical for other countries at all
three grade levels. This is true even for science texts devoted to
particular topics, such as earth science or physical science.
Mathematics curricula in the United States consistently cover
far more topics than is typical in other countries. In science, the
tendency toward breadth is similar, though less pronounced.
.
Topics remain in both the mathematics and science curricula
for more years in the United States than in all but a few other TIMSS
countries. The U.S. practice is to introduce many more topics than
do other countries in grades one and two and then to repeat these
4William Schmidt served as both the principal investigator for the curriculum
study and the national coordinator for the U.S. portion of the achievement study. He
also served as the project director for the Survey of Mathematics and Science Oppor-
tunities (SMSO). This study, conducted in advance of TIMSS, produced a set of
classroom observations in six countries that were designed primarily to identify
important themes and issues to be explored in the TIMSS background questionnaire.
His presentation drew on all of these sources.
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
9
OCR for page 10
topics through grade seven. Schmidt emphasized this point by noting
that although new elements called for by science standards have gen-
erally been added to the curriculum, little has been removed to make
room for the new.
American teachers, Schmidt argued, are sent into their classrooms
with a mandate to teach using curricula that reflect few decisions
about priorities, are fragmented, and are poorly integrated with one
another. Teachers, he said, are armed with textbooks that are simi-
larly laden with a jumble of topics. The curricula are, in his words, "a
mile wide and an inch deep." How do teachers handle this situation?
Schmidt argued that the instructional decisions made by U.S. teachers
mirror the inclusive approach of the tools they are given. Teachers
cover more topics, he suggested, but spend less time and emphasis on
each than do many of their international counterparts. Instead of
"telling a story" about a particular topic, allowing enough time for
students to learn it and move on, he argued, U.S. teachers tend to keep
reintroducing topics that have not yet been mastered.
Schmidt concluded that the U.S. educational vision is splintered
because the U.S. system has many actors and is characterized by "dis-
persed control," as Richard Elmore later put it. For Schmidt, this
system is responsible for the seriously inadequate sets of curricula
currently in use. The incoherence of the curricula, he argued, has
impeded student learning.
The Three-Country Qualitative Studies
Germany, Japan, and the United States participated in additional
studies, sponsored by the United States, in order to augment their
understanding of the achievement results. These studies, a videotape
analysis and a set of case studies, were devised to explore both in-
struction and the cultural contexts within which the learning and teaching
of mathematics take place. They involved methodologies rarely used
in conjunction with large-scale assessments of achievement, and, in
the case of the videotape study, of technology developed specifically
for TIMSS. James Stigler and Harold Stevenson, the principal re-
searchers for the videotape study and the case studies, respectively,
each described their methods and some key findings.
Videotape Study
The primary goal of the videotape study was to capture and then
analyze entire mathematics lessons taught to a subsample of the Popu-
lation 2 (grades seven and eight) students. Lessons were taped in a
total of 231 classrooms across the three participating countries. Teachers
were asked to make no changes in their normal classroom routines for
the videotaping sessions. Standardized camera procedures and other
protocols were developed for the data collection. The thousands of
10
LEARNING FROM TIMSS:
OCR for page 11
hours of tape were digitized, and computer software was developed
for analyzing them. Thus it has been possible for researchers to scan
quickly through the material on a computer and to search it in various
ways.
In addition, the tapes were transcribed and translated and then
coded for the occurrence of various events, teaching strategies, and
content elements. The coding made it possible for researchers to
analyze the lessons quantitatively and to explore such issues as amounts
of time spent on seatwork and classwork, discussing and doing home-
work, and non-lesson activity. The tapes were also analyzed by math-
ematicians for mathematics content.
In addition to making possible the exploration of questions about
teaching practice, as well as specific questions raised by data from
the achievement tests and the background questionnaires, the video-
tapes have two other important uses. First, as symposium partici-
pants who watched just a few short segments emphasized, the oppor-
tunity to observe a lesson on tape is far more powerful than any
verbal description can be. It is clear that the tapes themselves, as
well as the experience gained in collecting them, will be an extremely
valuable resource for teacher training, as well as for research. Sec-
ond, the digitized tapes are a permanent, unchanging resource. Fu-
ture research can be conducted using these tapes as a record of teacher
practice at a particular time, as research questions change.
Apart from the interesting technical issues Stigler and his team
faced, the videotape study produced some interesting conclusions about
variations in teacher practice among the three participating nations.
The report on the study had not been released at the time of the
symposium, but Stigler discussed several of its key findings. Perhaps
most important was Stigler's conclusion that the majority of prescrip-
tions about teacher practice that have been generated by the research
community in recent years have not been implemented in U.S. class-
rooms. Stigler argued that the relatively large-scale videotape study
has made it possible for the first time to look at what teachers are
actually doing in the classroom and to compare that with their verbal
descriptions of what they believe they are doing.
Citing the notion of problem solving, for example, a traditional
mathematics skill that is carefully redefined in the National Council
of Teachers of Mathematics (NCTM) standards, Stigler pointed out
that the understandings teachers and others have of what it means in
practice vary to an alarming degree. He described a lesson he had
observed, in which students solved a series of traditional word prob-
lems as a group. Their teacher had spoken enthusiastically about the
"amazing problem solving" the students were doing, believing that
she had fully responded to this aspect of the revised standards.
Stigler made the further point that major shifts in education policy
often occur without the benefit of any, or sufficient, data about the
extent to which the current policy has actually been implemented in
the classroom. This point is relevant to a question that many have
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
11
OCR for page 12
asked about TIMSS whether TIMSS achievement results could be
seen as a measure of the impact of the NCTM standards, which were
published in 1989, on student learning. Stigler's conclusion from the
videotapes is that the question is moot since the NCTM reforms have,
by and large, not been implemented in U.S. classrooms.
Stigler also reviewed some of what the videotapes revealed about
differences among the three nations. He noted that the sample sizes
chosen, 100 teachers for Germany, 81 for the United States, and 50
for Japan, partly reflected expectations about how much teaching styles
were likely to vary within each country. Surprisingly, Stigler found
that teachers in both Japan and the U.S. were remarkably consistent.
In general, Stigler's portrait of typical approaches to lessons in Japan
and the United States (his presentation focused on these two nations)
is likely to cause concern in the U.S. education community, and that
impression was strongly reinforced by the videotapes he showed.
The Japanese lesson showed a teacher who pushed his students to
grapple with a series of problems and to come up with alternative
solutions. The teacher communicated respect for his students' abili-
ties to cope with challenging material, and he guided the students
skillfully from the alternative solutions to a more general understand-
ing of the concept the lesson covered. In contrast, the U.S. teacher
seemed to lead his students by the hand through an explanation of a
concept, and he telegraphed his expectation that the students would
have trouble applying the concept in a challenging problem by warn-
ing them repeatedly about a particular problem as they began their
seatwork. Then, before they had had time to attempt that problem, he
stopped them and led them through it step by step. The U.S. lesson
was also interrupted more than once, both by conversation about school
schedules and other issues unrelated to the lesson and by an announcement
over the public address system.
These two excerpts were chosen by Stigler to represent what he
and his team had judged to be typical of the lessons he saw in the two
countries, and they raise issues that are familiar to many in the policy
and research communities. For Stigler, the videotapes from Japan and
the United States painted a consistent picture of two different ap-
proaches to teaching. He noted that the questionnaires administered
to the teachers who participated in the videotape study (these were
different from the questionnaires administered with the achievement
tests) revealed very different expectations for the outcome of a lesson:
70 percent of Japanese teachers reported that their goal was to get the
students to understand a concept; similar percentages of U.S. (and
German) teachers reported a goal of getting students to be able to do a
certain kind of problem. In Stigler's view, the Japanese lessons gen-
erally "tell a story" and provide students with the opportunity to struggle
with and explore the concept the teacher is presenting. In contrast,
the videotapes show relatively less development of concepts in the
U.S. lessons, which Stigler characterized as focusing on short-term
goals.
12
LEARNING FROM TIMSS:
OCR for page 13
To support his conclusions, Stigler explained a few of his spe
. a. I. .
ClilC IlnC .lngS:
.
While the proportions of time spent on incliviclual work ant!
work as a class are roughly the same in the two countries, Japanese
teachers tend to switch between the two much more frequently than
do U.S. teachers.
.
The U.S. teachers pay far more attention to homework than
clo their Japanese counterparts, allotting significant chunks of class
time for going over previous homework or allowing students to begin
new assignments. leaving relatively less time for instruction.
.
a, , a,
The U.S. lessons were interrupted by non-mathematics-relatec}
activities significantly more frequently than were the Japanese les-
sons. This fincling reinforced for Stigler the sense that in Japanese
society the lesson is regarclec! as a coherent, sustained inquiry into a
topic while in the U.S. it is regarclec! more as an episode or a practice
session.
· Japanese teachers generally focus on just one topic cluring
each lesson; U.S. teachers average close to two topics.
The participants' responses to the brief videotape excerpts were
extremely lively, ant! many remarkoc! on how convincingly the ex-
cerpts seemec! to illustrate particular arguments about teaching prac-
tice. Some of the issues raiser} both in the papers preparer} for the
symposium ant} by participants about ways of using ant} unclerstanci-
ing this kind of data are explorer} below ("Critiques ant} Methoci-
ological Issues").
Case Studies
While the primary focus of the videotape study was on teacher
practice, the case studies concluctec} by Harold Stevenson explorer} in
cletail the contexts that shape the experiences of students ant} teach-
ers. Like other parts of TIMSS, this stucly was clesignec! to provide
data to help account for some of the variations in student perfor-
mance, in this case by examining contextual influences. Previous
studies have shown that differences in curriculum ant} education structure
can provide insights about performance, but other kinds of informa-
tion are also neecleci. How do teachers in different places think about
teaching, learning, ant! curriculum? How have they been preparer!
ant! what kinds of support clo they receive? What factors in ant! out
of school affect students' motivation to learn? What are students'
attitudes about mathematics ant} its value?
While the contexts that shape learning can be explorer} through
written questionnaires, the case studies were an opportunity to make
cross-cultural comparisons in far greater detail ant! to investigate subtler
issues than a coclec! questionnaire could permit. Through this project
researchers intenclec! to produce thorough analyses case studies of
eclucation-relatec} factors in three distinct cultures, the United States,
RESULTS OF THE THIRD INTERNATIONAL MATHEMATICS AND SCIENCE STUDY
13
OCR for page 14
Germany, and Japan. The studies were structured around three basic
issues. The first was how content and performance standards in the
three countries compare. Through this comparison, researchers hoped
to explore the ways in which each country deals with individual dif-
ferences among students. The second was the role of school in ado-
lescents' lives. The last was the ways that training, certification, and
support for teachers' continuing professional development affect their
working lives.
Stevenson began by explaining that the study was a sort of hybrid,
devised for TIMSS, between the methods of anthropological ethnog-
raphy and the interview approach characteristic of psychology. The
result was what he termed "a descriptive study" "a description of
what you would find if you were in these particular cultures." The
basic plan was to identify and train individuals who were familiar
with each of the three societies, fluent in the requisite language, and
skilled in observation and interview techniques, and to send them into
the field to collect information. With the help of country experts,
sites were chosen that would broadly reflect national characteristics,
and researchers assigned to each country spent 2-3 months collecting
data.
The researchers spent the bulk of their time interviewing parents,
students, and teachers and observing classroom lessons. They visited
homes, schools, and education ministries. The result was hundreds of
hours of audiotape, which was transcribed and translated. As in the
videotape study, the material was entered into a computer and coded
so that researchers could search it efficiently, but the data were not
analyzed statistically; rather, they were synthesized into detailed de-
scriptions, organized around the explicit questions that guided the
study.
Like the report on the videotape study, the case study reports had
not been released at the time of the symposium, but Stevenson high-
lighted some of the insights that have emerged. One important focus
of his presentation was on ways in which detailed knowledge of cul-
tural contexts can significantly alter discussions about a particular
issue. His choice of an example homework was inspired by his
concerns about the ways in which symposium participants had dis-
cussed the relationship between homework and achievement results.
He noted that in Japan there are four possible translations for the
term, none of which corresponds to our notion of the word. The
Japanese terms describe a variety of activities one might do outside of
class study, work on practice questions, or do an assignment, for
example. They reveal that ways of categorizing such activity differ in
the two cultures. To further illustrate the point, Stevenson noted that
the amount of homework done by German students varies signifi-
cantly, depending on the type of school they attend. Consequently, a
mean for homework done in Germany would have very little value. It
is only through interviews, Stevenson maintained, that researchers were
able to discover what kinds of out-of-school studying students in each
14
LEARNING FROM TIMSS:
Representative terms from entire chapter:
videotape study
