International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies (1995)

Purpose The establishment of U.S. national goals, the push toward better means of assessment, and efforts to set standards for school science have led to a great interest in defining world-class standards in mathematics and science achievement of precollege students. One persistent question is the achievement expectations for students preparing for university admission in science or mathematics and/or planning to major in these fields. Therefore, the National Center for Improving Science Education will analyze the current mathematics and science tests given to students seeking university admission in several industrialized countries.

The aim is to develop more knowledge about how mathematics and science tests intended for the most accomplished students differ across countries, what can be inferred from the analyses of such tests about learning standards set for students in these countries, and how U.S. mathematics and science tests can be improved to match the standards of other countries. The National Center staff believe that their approach to this work not only will inform U.S. policy but will contribute to international understanding of educational expectations embodied in tests and the place of testing in national education policy.

Organization and Management The project is guided by an advisory panel that reviews drafts of the chapters and of the complete report and helps to refine dissemination plans. The study is taking place over a 28-month period (September 1992 - January 1995), during which an advisory panel was formed and experts contacted. Other activities include establishing a structure for analysis and reporting; selecting, collecting, and translating tests; analyzing tests and writing findings in a final report; and implementing a dissemination plan.

Participants Advanced-level examinations in mathematics and science (biology, chemistry, physics) have been obtained from France, Germany, Great Britain, Israel, Japan, Sweden, and the United States.

Sample The populations for tests selected and analyzed closely match Population 3b, the “mathematics and/or science specialist” population defined for TIMSS.

Procedures and Summary of Content The scope of work of the International Comparative Study of Mathematics and Science Tests complements the work of the IEA Third International Mathematics and Science Study (TIMSS). The science and mathematics frameworks developed for TIMSS were augmented and used to analyze the content and performance expectations of the selected tests. To the extent possible, TIMSS participant survey questionnaires will be used to gather data on what sorts of students are taking the tests to be analyzed, how the tests are used in each of the countries, and other contextual factors. The test analysis project is being carried out in conjunction with TIMSS leadership; there is an overlap in personnel; and all countries selected to participate in this study are participating in TIMSS.

Data Collection and Analyses The International Comparative Study of Mathematics and Science Tests is working closely with the TIMSS International Coordinating Center to collect the tests.

University of British Columbia graduate students are translating tests that were not provided in English.

Examinations collected for analysis are in the fields of mathematics, physics, chemistry, and biology. For each country, each field, and pertinent type of test, the two latest years of tests available will be collected, in order to have an adequate representation of the domain of knowledge and performance expected of students. Expected responses or response protocols and scoring rubrics is included.

In 1994 an international team of researchers assembled by the National Center for Improving Science Education analyzed national examinations in mathematics and science (biology, chemistry, and physics). Research team members and study advisors, drawn from the United States and other countries, included eminent experts in science and mathematics education. The exams compared were end-of-secondary school tests, given to college-bound science and mathematics students, from seven countries.

Analysis of the tests in mathematics and in each of the science fields, used the TIMSS mathematics and science curriculum frameworks (mathematics, John Dossey; biology, Pinchas Tamir; physics, Kjell Gisselberg; chemistry, Dwaine Eubanks). Each of these experts will produce a chapter on the results of the test analyses across countries together with an overview and conclusions regarding the achievement and performance expectations for advanced students in the specific subject. A summary chapter will be added synthesizing the separate reports on each of the fields and providing overall conclusions. Analyses and draft chapters will be reviewed by prominent senior researchers (Paul Black, physics; Angelo Collins, biology; Dorothy Gabel, chemistry; Curtis McKnight, mathematics; and Jack Schwille, comparative education). Two major components of the study are item-by-item categorizations of the tests' (1) topics and (2) expectations for student performance. To inventory these two characteristics, researchers developed supplemented versions of the Science and Mathematics Curriculum Frameworks from the Third International Mathematics and Science Study (TIMSS). Consequently, researchers had between 50-75 subject matter categories available (per subject), nested in three hierarchical levels, to describe an item's topic.

Researchers used multiple categories to describe an item's topic as needed. All researchers also used a second dimension of the TIMSS curriculum frameworks to describe each item: expectations for

student performance. Twenty categories were available, including, e.g., understanding of simple information, developing explanations using scientific principles, and solving quantitative problems.

Many other features of the exams were compared. Nine categories were used to document the formats of questions, e.g., multiple choice questions (3, 4, or 5 response options), open-ended (types of short-answer questions, types of extended-answer questions), and practical tasks. Researchers also inspected the wording of items (e.g., whether questions were stated negatively or positively) and the demands of problem-solving questions (e.g., whether questions merely required set-ups for solution or completely worked solutions.) Finally, more global exam features were noted such as length of exams and kinds of options included (or not) for students to select questions.

In late 1994 the report on detailed comparative analysis of exams will be published. Researchers compared three main characteristics of the tests: topics covered, expectations for student performance, and types of questions. The Center's international team also compared the overall lengths and components of the tests, the amount of choice students have in selecting questions, the complexity of the topics covered, and wording devices used in constructing questions. There is only one exam from each year in the report, which is too few exams for determining the range of topics that national exams typically assess. A full range of topics would require inspection of several years of exams; however the Center's analysis of exams from two years will go a long way toward showing the range of topics included in the exams of other countries.

Exam topics in the Center's report are described using detailed categories that the Center helped to develop for a the Third International Mathematics and Science Study (a large-scale international study involving a large number of countries). The topic of each exam question is noted using three levels of increasingly detailed topic categories; then each exam 's topics are summarized, making comparisons among exam topics possible. The report addresses what kinds of processes students must use to answer exam questions: whether students must give information by rote, or solve quantitative problems, or give explanations based on scientific principles. Categories known as “student performance expectations,” which the Center helped develop for the TIMSS study, were used to describe student performance. Five main categories (understanding, solving problems, using tools and procedures, investigating and communicating) and 15 sub-categories were employed. The report extensively discusses difficulty of exams and considers many factors that should be brought to bear when making comparisons between the “difficulty” of different countries ' exams.

In May 1994, the National Center and the American Federation of Teachers copublished What College-Bound Students Abroad Are Expected to Know About Biology. The volume contains university entrance exams from England and Wales, France, Germany, and Japan, and a U.S. Advanced Placement biology exam, as well as comparative descriptions of those nations' examination systems. Release of this report drew considerable media attention. It is recommended by the National Science Teachers Association and the National Association of Biology Teachers.

What College-Bound Students Abroad Are Expected to Know About Biology. National Center for Improving Science Education and National Federation of Teachers. (May 1994) (Similar publications on chemistry and physics are proposed.)

The end product will be a monograph released late in 1994 that should be of considerable interest to policy makers and scholars concerned with standards in mathematics and science and with testing. A number of diverse strategies will be used to disseminate the results of this study as widely as possible.

Funding Funding is provided by the National Science Foundation.

Senta A. Raizen, Director

The National Center for Improving Science Education

2000 L Street, N.W., Suite 603

Washington, D.C. 20036

telephone: 202/ 467-0652

facsimile: 202/ 467-0659

e-mail: sentar@gwuvm.gwu.edu

Ted Britton, Project Director

The National Center for Improving Science Education

2000 L Street, N.W., Suite 603

Washington, D.C. 20036

telephone: 202/ 467-0652

facsimile: 202/ 467-0659

e-mail: tbritton@gwuvm.gwu.edu

National Center for Improving Science Education

1992 International Comparative Study of Mathematics and Science Tests: Project Summary.

1994 Content Analysis of National Science and Mathematics Examinations from Seven Countries: Proposal Summary. Symposium Proposal for American Educational Research Association, 1994.

National Center for Improving Science Education and National Federation of Teachers

1994 What College-Bound Students Abroad Are Expected to Know About Biology. Defining World Class Standards Volume 1. National Center for Improving Science Education and National Federation of Teachers, Washington, D.C.

Raizen, Senta

1992 Content Analysis of National Science and Mathematics Examinations from Seven Countries. Presentation to Board on International Comparative Studies in Education. October.

******

NOTE: This study summary was reviewed and edited by Ted Britton at the National Center for Improving Science Education in Washington, D.C. on May 31, 1994.