A Collaborative Agenda for Improving International Comparative Studies in Education

If the situation is to be improved, the tasks of establishing a more coherent and effective worldwide system for cross-national research and data collection in education must be shared among many organizations in many countries. This document does not presume to set the agenda for international research, or even to indicate the priorities for such research-- activities that would be appropriately carried out only within the framework of a worldwide system for cross-national education studies. This document was prepared to stimulate interest and a willingness to engage in establishing such a system by suggesting a vision of what international comparative research and data collection might produce in the next 20 years.

By the year 2010, this vision calls for those with an interest in learning about education in other countries to have access to the following sources of data, information, and analysis:

• A set of widely accepted and widely understood indicators of the status and conditions of education in various countries; [1]
• Syntheses of empirical research throughout the world, bringing such research to bear on broad comparative questions of wide interest;
• Specialized comparative monographs on questions of importance and interest to all educators;
• Documentation of teaching practices and school organization in a form that will be stimulating and useful for discussion and study by specialists and nonspecialists, including, for example, videotape archives of a wide range of classroom practice;
• Widely distributed publications of comparative information on issues of public policy, written in nontechnical language and responsive to the interests of journalists and the public;
• Routine translation, distribution, and use of primary documents, such as textbooks and syllabi, when these documents are sufficiently distinctive and important to be of interest to educators outside the country of origin; and
• Archives necessary to strengthen the methods used in cross-national studies, including both technical reports on what works methodologically and banks of instruments, items, and other applicable research elements and models for a variety of cross-national studies.

Nevertheless, much of education research is parochial, uninformed by the comparative knowledge that cross-national education research can provide. Researchers should be encouraged to develop broad and in-depth knowledge about cross-cultural similarities and differences and similarities in the way people learn. They should also undertake studies that examine the context of and participation in formal and informal education at all levels. Research is also needed to generate more narrowly focused knowledge that speaks to pressing problems of practice, such as identifying pedagogies and schools that best help children at risk to avoid becoming dropouts or knowing the role of algebra in curricula designed for today's students and how they learn it best.

It would be important, therefore, to seek ways to engage education researchers, practitioners, and consumers in the search for international comparative understanding of what education is and how it should be judged. Researchers should capitalize on what can be learned from a natural worldwide laboratory of education systems, which over generations have come to embody many important differences in opportunities to learn, students' desire to learn, curricular structures, and teaching practices. Addressing these differences involves making inferences about what different cultures and organizations around the world consider to be excellent outcomes and excellent practice. Education policy makers should focus on success wherever they find it and could also learn from what does not seem to work. It is especially important to seek data over time in order to examine trends and the impact of changing contextual conditions. In addition, the collection and sharing of the artifacts of education (e.g., textbooks, children's writings, teachers' journals, and videotapes of classrooms) should be encouraged so that discussions of education in other places and at other times can give the users of such artifacts a better sense for the social texture and cultural intricacies that are so central to knowledge of teaching, learning, and human development.

The balance of this section focuses on the two broad types of international comparative knowledge about education; each type satisfies different needs for information. First is the knowledge needed to address issues of policy and practice currently on the public agenda. Approaches in this category focus on problems for which researchers have arrived at a working definition and for which they seek alternative solutions to compare and evaluate. Second is knowledge without reference to a particular current problem or issue--such knowledge is gathered to enhance general understanding in the long term.

An important purpose of cross-national research is to respond to particular policy needs. For example, an issue of particular importance to U.S. policy makers at present is whether all our students have been given adequate opportunities, appropriate standards, and justifiable expectations for learning, relative to what is done in other countries. In this respect, it is important to examine out-of-school opportunities as well as those provided by schools. Questions range from "What can be expected of our best students?" to "What can be done about students who are poorly prepared and poorly motivated?" In short, what can be done to make sure that all students truly have adequate opportunities to learn?

Such policy issues can be addressed in three ways:

• Tracking progress within and among nations;
• Comparing the quality of education offered while explicating and attempting to understand its context; and
• Policy-driven comparative studies of education institutions and practices.

Tracking Progress

The lack of an adequate system of education indicators to inform education policy making has become increasingly apparent. Data are not collected regularly, systematically, or with enough coordination either to satisfy natural curiosity about education systems around the world or to answer the questions of researchers and policy makers about changes over time in education in a variety of countries. Trend data are needed on many aspects of education. We illustrate with three types of trend data that are needed:

What children in other countries are learning. International comparisons provide comparisons beyond the limits of U.S. national experience. For example, recent studies have shown that students in other countries study some mathematics topics at an earlier age than students in the United States and that consequently it may be feasible to modify U.S. curricula accordingly. In the absence of absolute standards for education, comparisons help policy makers set realistic standards. Periodic collection of data from a diverse group of countries can serve both descriptive and monitoring purposes (Bradburn and Gilford, 1990).

Factors that may be related to school achievement. Information from different countries on policy-manipulable variables that are associated with student performance can identify potential policy improvements. Such variables include the time that children in other countries spend studying important subject matters such as mathematics, science, mother tongue, history, and foreign languages; the knowledge and expertise teachers bring to their work; and classroom size.

School finance data related to the delivery of education. Education expenditures such as teacher salaries, expenditures for in-service and other continuing education for teachers, expenditures on textbooks and other instructional materials, expenditures for special services, and class size (a determinant of the total expenditure for teacher salaries) can reflect various education policies. Education policy makers find it useful to compare trends in these indicators for the United States (a reflection of U.S. education policy) with trends in these expenditures for various countries that have education policies different from those in the United States, although the data must be used with caution because of the difficulties in obtaining comparable financial data.

Not only should national averages and other measures of central tendency be investigated, but also indices and graphic representations of variation within and between countries and trends over time as they become available.

International agencies have begun to provide information to answer questions about some aspects of education. For the past three decades a nongovernmental consortium known as the International Association for the Evaluation of Educational Achievement (IEA) has endeavored to fill a major gap by providing cross-national data on educational achievement, augmented by extensive information concerning the curriculum, classroom process, and other contextual information necessary to its interpretation. IEA studies have covered a broad range of topics: mathematics, science, reading literacy, written composition, literature, English as a foreign language, French as a foreign language, civic education, classroom environment, computers in education, and early childhood care and education. IEA has not identified or attempted to deal with all the important issues that should be addressed by indicators, nor has it been sufficiently well financed or well organized to set up regularly scheduled international assessments in all important school subject matters. Recently, however, IEA has released plans for a schedule of 4-year cycles of studies that provide for reports: a cycle for mathematics and science starting in 1995, a cycle of special studies such as preprimary education or computers in education starting in 1996, a language cycle starting in 1996 and alternating between second or foreign language and reading literacy, and a cycle starting in 1997 alternating between civics and the arts.

The Organization for Economic Cooperation and Development (OECD) has organized an extensive effort, growing out of preparatory conferences held in 1987 and 1988, to plan for the development of comparable education indicators across its member countries. Six networks were formed to deal with student achievement, education and labor market participation, features of school systems, attitudes and expectations, student flows, and costs and resources. Participation in these networks ranges between 6 and 20 countries. In 1992, OECD published a first set of international education indicators (OECD, 1992) developed by the Centre for Educational Research and Innovation (CERI). It provides comparative information for the OECD countries for three clusters of indicators: the demographic, economic, and social context of education systems; costs, resources, and school processes; and outcomes of education. Future publications of OECD indicator data, which are expected to be published at about 16-month intervals, will establish important trend information on OECD education systems for use by policy makers.

The United Nations Educational, Scientific and Cultural Organization (UNESCO) also collects country data on key aspects of education in over 160 countries. During the 1950s and 1960s, UNESCO issued a periodic publication, A World Survey of Education, which provided statistics on the state of education throughout the world. In 1992, UNESCO published World Education Report 1991 (UNESCO, 1992), the first in a new biennial series that will present major trends and current policy issues in education. Focusing on basic education, the 1991 report describes world education growth since 1970 and includes information on demographic trends, adult literacy, participation in formal education, teachers, and finance. The report focuses on basic education and on continuing challenges in that area. Looking toward the future, the report identifies two major issues: teachers and teaching (e.g., the need for additional teachers, recruitment, and training) and the assessment of students' learning achievement (e.g., school-based assessment, changes in assessment practices). The report includes education indicators for the years 1980 and 1988, thereby providing measures of change in education characteristics, for over 160 countries on a number of topics: population and gross national product; literacy, communications, and media; school entry and participation; enrollment ratios and internal efficiency in first-level education; enrollment ratios in second level education; teaching staff in preprimary, first- and second-level education; enrollment ratios and teachers and students by broad field of study in 1988 in third-level education; public expenditure on education and private enrollment; and current public expenditure on education.

These OECD and UNESCO reports are welcome additions to education information. In planning for future indicator sets, a 1991 report by the Special Study Panel on Education Indicators for the U.S. National Center for Education Statistics (NCES, 1992) is particularly relevant. The five principles advocated in this report and listed below were developed for national indicator sets, but they could be applied to cross-national sets as well:

• Indicators should address enduring issues. Although a model for indicator development oriented to education goals (i.e., define goals, specify objectives related to the goals, and develop indicators to measure progress toward the objectives and goals) could be used, it would lose its value over time as goals change. Researchers should assess what they think is important, not settle only for what they can measure.
• The public's understanding of education can be improved by high-quality, reliable indicators.
• An effective indicator system must monitor education outcomes and processes wherever they occur (not just in schools).
• An indicator system built solely around achievement tests will mislead the people in the United States.
• An indicator system must respect the complexity of the education process and the internal operations of schools and colleges (National Center for Education Statistics, 1991:19-21).

Moreover, for use in informing policy and practice, sets of indicators are more useful than individual indicators, and these sets must be developed for a specific context, form, and level of education.

Comparing Quality While Understanding Context

The interested layperson naturally wants to know how the quality of education in the United States compares with that of other countries. Often this concern is addressed in ways that pay scant attention to the complexities of education and the particular contexts in which children are educated. Cross-national studies in education have attempted to deal with this dilemma by developing methods and measures that avoid comparing the incomparable. Thus, for example, the concepts and measures developed for IEA studies include: the intended curriculum (what the curriculum specifies), the implemented curriculum (what the teacher actually taught), and the attained curriculum (the student performance on a test of content). This refinement represents one way to achieve more comparability in the explanation of achievement outcomes (although much more improvement is called for in the implementation of these curriculum measures). In general, a great deal of additional work on explanatory variables is needed to increase the comparability of education processes and outcomes.

Still another approach to dealing with the difficulties of comparability is illustrated by earlier IEA research that focused on how to adjust results to take account of more and less selective secondary school systems to avoid comparing the elites of one system with a larger mass of students in another. The goal of these studies was to help policy makers consider the advantages and disadvantages of a more or less comprehensive secondary school system. The findings were used to support policy changes making a number of secondary school systems more comprehensive (e.g., Marklund, 1989; Bathory, 1989).

Curriculum and selectivity differences are not the only issues in understanding the context for cross-national comparisons. Similar issues include differences in language spoken, religion, laws, implements (e.g., tools, utensils, instruments) used, and values held, as well as attitudes toward testing that might influence motivation. Differences in cultural context can affect what is taught and when it is taught. Research is needed on the effects of these cultural differences on student achievement and the development of a theory explaining the contextual difference among nations.

Policy-Driven Comparative Studies of Educational Institutions and Practices

U.S. national education goals for the 1990s have provided strong motivation for continued interest and discussion of cross-national comparisons in mathematics and science (U.S. Department of Education, 1991). Additional motivation has been provided by the Goals 2000: Educate America Act (introduced in the U.S. House of Representatives and the U.S. Senate), which formally establishes the education goals and calls for a National Education Goals Panel and a National Education Standards and Improvement Council, which would develop model standards for student performance, curriculum, and school quality (H.R. 1804 and S.R. 846, 1993). Several implicit causal assumptions underlie public and political interest in cross-national studies of mathematics and science: (1) increased and improved curricular emphasis on mathematics or science in precollegiate education for all students will result in greater achievement for a greater number of students; (2) this will result in a greater number of better prepared students entering the natural sciences and engineering; and (3) this will in turn lead to a greater number of more productive scientists and engineers in the labor force. A final assumption is that more and better scientists and engineers will increase economic growth or productivity. Each of these premises has its critics, and each could be more systematically evaluated. At a minimum, conferences should be organized to ensure that all reasonably well-informed parties with opinions on such issues are heard, reports should be issued, and, if possible, strategies should be developed to keep the contending parties from speaking past or ignoring one another's point of view.

However, a focus on mathematics and science is not sufficient; other subjects are also important. It is critical to ensure that the complexities of education, including its variation across contexts and multiple content areas, are taken into account. Virtually every policy issue can be illuminated through study of how policy and practice vary among countries. For example, recent cross-national comparisons highlight gender differences and similarities at both the individual and societal levels of analysis. These studies can facilitate our understanding of the implications of the gender gap in academic achievement (e.g., they allow us empirically to evaluate arguments that suggest that more centralized education systems are more equitable and/or more efficient). These studies also help us understand the effects on economic growth of the expanded access of women to varying levels of education. Data from the studies can be used to evaluate arguments that suggest that the undereducation of women negatively affects economic growth (King, 1990).

The range of issues ripe for comparative inquiry includes questions of finance, decentralization versus centralization, the professionalization of teachers, the need for formal programs of teacher education, the effects of education on productivity, the effectiveness of measures whose goal is to enhance education opportunities for minority populations, the value of bilingual education, differing mixes of public and private schools (including religious schools), and even the feasibility of emphasizing moral education in public schools. Although much has been said about the world's constituting a natural laboratory, in fact the availability to U.S. audiences of current, relevant, and in-depth information on policy issues in other countries remains far too limited. Although it is true that studying education in other countries will not tell us what we ought to do, such study will help us to become clearer about what we value and the policy options for putting these values into practice.[3]

What does it mean to pinpoint areas in which cross-national differences are important to education policy and practice and to attempt to provide reasonably persuasive explanations for such differences? Heretofore, this goal has meant an attempt in cross-national surveys to estimate how much education outcomes are affected by factors controlled through education policy. This effort has been motivated by the idea that comparisons of achievement levels are not meaningful unless one can, first, identify the educational inputs and processes that contribute to observed outcome differences between countries; second, make some estimate of the contribution of each educational input to realized outcome levels, and third, consider how these effects vary by context.

But surveys by themselves are not adequate. They have focused too much on narrow measures of achievement, neglecting other important outcomes. They are not very sensitive to contextual differences, and as a result it is often very difficult to give plausible explanations for observed differences. The complexities and difficulties of acquiring a more adequate understanding of education in other countries and its possible implications for the United States can be illustrated by looking at Japan. Since progress has been made in U.S. understanding of Japanese education, let us look carefully at how this understanding has evolved, at what has been learned, and at the gaps and ambiguities that remain in understanding. Turning back to the increasing exposure to Japanese education over the past 20 years, it becomes clear that there are different phases, which have gradually led to more, though still inadequate, comparative understanding.

During the 1980s, policy makers became aware that achievement levels on conventional multiple-choice tests in mathematics and science were higher for Japanese than for U.S. students at certain grade levels. For example, the IEA Second International Mathematics Study, which gathered data in the 1981-82 school year, reported that Japanese eighth-grade students consistently scored higher than students in other countries in arithmetic, algebra, geometry, and measurement, whereas U.S. students scored in the middle of the group of countries on arithmetic and algebra and below the international average on geometry and measurement (McKnight et al., 1987:20-22) In science, in grades 5 and 9, Japanese students also scored at the top of the group, although at grade 12 their scores were lower (International Association for the Evaluation of Educational Achievement, 1988:7, 26-27, 32-36, 48-54). This awareness was accompanied by debates over the significance of and reasons for such results. There was much disagreement over whether these measures were meaningful when taken in isolation from other evidence about education outcomes. These debates stimulated public interest, yet they were initially largely uninformed by further empirical research on these issues.

Gradually, however, there have been more focused and data-based studies that were intended to provide more in-depth understanding of Japanese practices (e.g., studies by Cummings, Peak, Rohlen, Shimahara, Stevenson, Tobin, and others). As a result, relatively uninformed endorsement or rejection of Japanese practices has partially yielded to attempts to explain these practices more thoroughly within the Japanese cultural context. More attention has been directed to the many interactive factors that might influence a particular practice or outcome, such as the amount of time spent on a task or on homework, teacher quality, and parental beliefs about the relative importance of factors influencing success in school.

As such studies become more generally known, noncomparativists can be expected to cite these findings more frequently and to incorporate them in a more general understanding of mathematics education. Perhaps in the near future there will even be a sizable market for publications about how mathematics and science are taught in Japan as well as other topics.

To the extent that such writings bear fruit in explicating practices that are effective in Japan, there will inevitably be more debate about whether similar effects might be anticipated in the United States. Although the borrowing of educational policies and practices is much criticized for infringing on cultural values and indigenous practices and for its undeniable risks of unintended consequences, it can also positively influence classroom practices. It has been and will no doubt remain widely attempted in education.

The same principles, in many respects, apply to other countries and subject areas.

However, it is very difficult to carry out multivariate analysis of natural variation that is reliable and valid. For example, the high scores of Japanese eighth-grade students relative to those of the U.S. students in the IEA Second International Mathematics Study raised questions about the quality of precollege mathematics teachers in the United States. But, although poor student outcomes may be correlated with teacher quality in terms of training characteristics or classroom method, the cause may be due to factors unrelated to teacher quality, such as the curriculum, time spent on mathematics topics in school, and the influence of the home environment. The effects of all these factors on student achievement need to be understood before we understand the relative performance of Japanese and U.S. students. Although international studies have provided detailed information on student achievement in various subjects in many countries, the causal analyses of the comparative results have been inadequate. Advances in survey analytic techniques, such as hierarchical linear modeling, offer promise of addressing those weaknesses.

A second important purpose of cross-national research is the long-term development of knowledge. Scholars often gather knowledge for its own sake, sometimes to expand the knowledge base in their field and at other times to develop their capacity for grasping trends and advancing ideas. In the long run, this exploration enriches and expands their understanding of the world and its complexities. They make an intellectual investment, to be drawn on later, as they interpret their research findings and the results of other related investigations.

To underwrite such investments, it is important to collect cross-national data at societal levels over reasonably long periods of time. Such data facilitate the identification of worldwide, regional, and national trends and permit analysis of the sources and effects of cross-national variation in education organization, policy, and practice. Systematic cross-national inquiry, motivated by theory and carried out over time, is needed to discourage parochial folklore and inappropriate inferences rooted in a spatially and temporally limited vision of what education is as well as what it could or should be.

More specifically, long-term development of knowledge can affect comparative inquiry in a number of important ways:

• Becoming more cumulative and theoretical by providing synthetic works and bringing about consensus on a cross-national system of indicators, which in turn will illuminate changes across time;
• Highlighting areas in which cross-national differences are important to education policy and practice as well as provide reasonably persuasive explanations for such differences;
• Stimulating the development of a comparative dimension within all major educational specialties (e.g., empirically based comparative education has had far too little to say about the realities of classroom practice);
• Communicating the state of comparative education knowledge more effectively than at present to nonspecialists so that the thinking and discourse of educators in other fields will, as a matter of course, be informed by international comparative studies;
• Drawing attention to potential weaknesses or strengths of education systems;
• Identifying models or practices of education in other countries that have rarely, if ever, been used in U.S. education; and
• Expanding the categories and solutions that researchers consider when they think about education problems and thereby contribute to the contextual understanding of education.

In designing and carrying out such long-term efforts, a better understanding will be gained not only of contextual variation in education policies, institutions, and practices and their effects on learners, but also of more fundamental aspects of human and social development, including:

• The developmental processes of children, youth, and adults and the implications of these processes for learning;
• Education issues related to gender, race, and social class;
• The importance of cultural values in shaping the identities for which children are educated; and
• The role of education in relation to subsequent social and economic stratification in adult life.

Knowledge of a nation's education system will reveal what a nation values in its people. Is education for some or for all? What cultural norms does the education system embody? Are assumptions made that some people are unworthy of education investment? What are the personal and national payoffs of formal schooling? Who are the intended beneficiaries of the system, in addition to students? Such issues are of paramount importance in considering how nations determine who is to be educated, what is to be learned, and how all this can be accomplished. The central role of teachers in the success of an education program should be studied, as should issues concerning their recruitment, retention, training, rewarding, development, and deployment. Equally important are issues related to formal and informal sources of control and autonomy.

2 Some critics have noted, however, that controlled experiments have the potential to improve the methodological sensitivity of education studies. Recommendations for greater use of experimental methods are discussed in a recent National Research Council report on bilingual education studies (Meyer and Fienberg, 1992). An exclusive reliance on correlational research designs and data analysis techniques is a methodologically limited outlook. Thus, it is important to continue to weigh the pros and cons of the arguments against controlled education experiments.

3 It should be noted that countries with decentralized education systems such as the United States can be considered to be a self-contained natural laboratory. Cross-state studies also have the potential to contribute to knowledge about the impact of education practices and policies on education outcomes.

NAS Home Page | NAP Home Page | Reading Room | Report Home Page