International Comparative Studies in Education: Descriptions of Selected Large-Scale Assessments and Case Studies (1995)

Purpose Increased need for information on education led the members of the Organisation for Economic Co-operation and Development to initiate the International Education Indicators Project in an effort to create a set of comparative international education indicators that represent the key features of education systems. The project was launched via two conferences, hosted by the government of the United States (1987) and the government of France (1988). The project went from the planning phase to the production phase following a conference on the project hosted by the government of Switzerland in 1991.

The International Educational Indicators Project is an effort of the Organisation for Economic Co-operation and Development-Centre for Educational Research and Innovation to develop a system of indicators for cross-national comparisons in education for the use of policy makers, consumers, and “third parties” like private industry. To achieve this purpose, INES is:

• developing, collecting, analyzing, and offering a preliminary interpretation of a set of key indicators for international comparisons

• providing a forum for international cooperation and the exchange of information about methods and practices of developing and using educational indicators for national policymaking and managing education systems

• contributing to evaluation methodology and practice to develop more valid, reliable, and comprehensive indicators, and to gain a better understanding of their use in policymaking.

The task of INES Project Network A - Educational Outcomes is to:

• Develop indicators of student performance outcomes.

• Develop a comprehensive analytic frame for educational achievement outcomes.

• Develop specific criteria and standards for constructing and evaluating indicators of educational achievement.

• Apply the frame, criteria, and standards to existing international surveys.

• Suggest how these principles can be used by OECD to obtain achievement data on a regular basis.

Organization and Management The INES Project is managed by the Centre for Educational Research and Innovation within the Organisation for Economic Co-operation and Development. The work of the project is being carried out by a consortia of countries comprising a Technical Group and four Networks:

• Network A: Educational Outcomes (chaired by the United States)

• Network B: Education and Labor Market Destinations (chaired by Sweden)

• Network C: Features of Schools (chaired by the Netherlands)

• Network D: Attitudes and Expectations (chaired by the United Kingdom)

Using a network structure as a basic organizing framework, the first Phase of the INES Project explored the feasibility of developing and reporting comparable indicators concerning the education systems of participating countries. The network structure had five domains: student flows, student outcomes, functioning of schools, costs and resources, and attitudes and expectations. Each network had a lead country that coordinated the contributions of members, supported the theoretical and technical work required, and produced the network report.

At the end of the first phase in 1989, Network 2, Student Outcomes, was split into two groups: Network A, Student Achievement Outcomes and Network B, Education and Labor Market Destinations. The United States was the lead country for Network 2, and continues in that role for Network A. Network A (now called Educational Outcomes) is responsible for the development and preparation of educational outcome indicators.

Phase 3 project organization continues along the same lines as the previous two phases, with the exception that the two former technical groups have been combined, and the leadership of the network on School Processes has been passed from France to the Netherlands. Scotland has assumed responsibility for the network investigating indicators of attitudes and expectations. Network A continues to be chaired by Gary Phillips of the National Center for Education Statistics in the United States.

Design Following an exchange of preliminary information on definition of fields, concepts, methods of work, and data availability, networks started working in January 1987 to establish a working plan for the definition of a limited set of indicators. These plans were then further refined and endorsed by a scientific advisory group, which in Phase 2 was reconfigured into a consultative group. In Phase 3 the consultative group was replaced by a policy review advisory group, chaired by the chairman of the CERI Governing Board. In Phase 2, networks and technical groups conducted further work with the aim of selecting some trial indicators, establishing data sources, and undertaking some experimental indicator calculations. The results, analyses, and requirements for further work are published in Education at a Glance.

Participants

Eighteen countries participated in Phase 1 of the INES Project Network A - Educational Outcomes: Australia, Belgium, Canada, Denmark, France, Germany, Italy, Japan, Luxembourg, The Netherlands, New Zealand, Norway, Portugal, Spain, Sweden, Switzerland, the United Kingdom, and the United States.

Seventeen countries participated in Phase 2 of the INES Project Network A - Educational Outcomes: Belgium, Canada, Denmark, Finland, France, Germany, Italy, Japan, Luxembourg, The Netherlands, New Zealand, Norway, Spain, Sweden, Switzerland, the United Kingdom, and the United States.

Countries registered for Phase 3 (1992-96) are:

Procedures and Summary of Content In Phase 1 the INES Project Network A developed a conceptual framework in which education outcomes were seen as a function of resources, environment (including demographic, economic, and social aspects), and structural processes. Work was organized around three areas: completion of secondary education, cognitive achievement and other assessment outcomes, and activity following completion of secondary education. A list of indicators in each area was generated, discussed, and refined.

Phase 2 INES Project Network A aims were to develop a comprehensive analytical frame for educational achievement outcomes; develop specific criteria and standards for constructing and evaluating indicators of educational achievement within the general template established for the INES Project; and apply the template, criteria, and standards to existing international surveys to test the extent to which existing data and procedures meet these criteria and standards.

The INES Project entered into its third and final phase at the beginning of 1993. This phase, scheduled to continue for five years, aims at establishing an organizational framework allowing for the regular production of a set of international education indicators. It includes continuation of ongoing conceptual work and regular publication of the calculated indicator set as Education at a

Glance. At the end of this phase it is anticipated that the indicators function will become part of the regular responsibilities of the Education Division of OECD.

During Phase 3 the second edition of Education at a Glance has been published (1993), and a completely revised edition will be published in 1994. A state-of-the-art report on the results achieved in implementing an effective international information system on education statistics will be an additional product. By the time the INES Project is completed in 1996, the Organisation for Economic Co-operation and Development and its Member countries should have succeeded in setting up an international database containing up-to-date and relevant education statistics and indicators; a fully computerized on-line network for collecting and disseminating the data; and cost-effective procedures for processing the data.

Data Collection and Analyses In Phase 2 INES Project Network A surveyed member countries in an attempt to identify existing data from national assessments or examinations that might serve as outcome indicators. The survey produced much valuable information about the variety of assessment structures and practices, and allowed some conclusions to be drawn about commonalities. It also demonstrated that it would be extremely difficult, if not impossible, to depend on data produced from these examinations or assessments to produce comparable indicators, as they vary widely in purpose, ages tested, subjects tested, and forms of testing.

Network A proposed a set of indicators (multiple comparison of mean achievement scores, international comparative distribution of achievement scores, learning/teaching ratio, and between school and between classroom variation in achievement scores) that can provide information about the productivity of the system relative to other countries, the effectiveness with which the curriculum has been learned, and how equitably achievement is distributed within a country. These indicators were calculated on a trial basis using data from the IEA Second International Mathematics Study and the mathematics portion of the Second International Assessment of Educational Progress. The results were published in the first edition of Education at a Glance.

In Phase 3 each of the INES Project networks and the technical group are involved in data assembly and preparation of indicators for the third edition of Education at a Glance. The technical group is piloting the use of a new data collection form for finance data. This form is also being used as the basis for a revised OECD/UNESCO/EUROSTAT pilot of the joint survey. This form is an outgrowth of the Expenditure Comparability Study conducted for the National Center for Education Statistics.

Current long-term activities of the INES Project include writing and editing chapters for several technical volumes that the project will publish in 1995, piloting data collections to measure cross-curricula competencies, and examining the use of national assessments for international comparisons. Each network is preparing books on their work to date and its implications for future international comparison work. These books are intended to compile and make more publicly available the methodological contributions achieved in Phase 3 of the INES Project.

The second edition of Education at a Glance was published in December 1993. It contains 38 indicators organized by 1) cost, resources, and school processes; 2) contexts of education; and 3) results of education. National Center for Education Statistics representatives to the project's networks and technical group were responsible for providing information about the United States and preparing the student outcome indicators. NCES is involved in developing ideas for indicators and data collection strategies for the indicators to be published in the third edition of Education at a Glance. These ideas include a survey on attitudes toward education, a description of the incorporation of

national goals into educational systems, and the measurement of the educational attainment of workers in different occupations and industries.

The INES Project attitudes survey is a public opinion survey that includes questions on subjects that young people study in secondary school, qualities that young people may have developed by the end of their compulsory schooling, the importance of areas for schools to emphasize in order to achieve their goals (e.g., career advice), how much responsibility the school should have (compared with the home) for the personal and social development of young people, the importance of certain decisions being made at the school level, and the respect for and earnings of secondary school teachers. About ten OECD countries will be participating in the survey.

For the second edition of Education at a Glance, Network A prepared indicators for reading literacy, mathematics, and science for 13/14-year-olds. Three types of indicators were prepared: comparison of means (multiple-comparison chart); distribution of scores; and between and within school variance. In addition, Network A prepared a brief cross-subject-based introduction to its section. As with indicators presented in the first edition of Education at a Glance, all indicators used the standards that Network A has adopted for its work.

Network A remains committed to calculating the indicators already developed, and the second edition of Education at a Glance presented an expanded set using science data from the International Assessment of Educational Progress study and indicators of reading literacy from the International Association of Educational Achievement Reading Literacy Study at two different age populations. Network A has also explored some new indicators of cognitive achievement, including attempts to measure the “value added” to student learning by schooling, and a measure of change over time.

At a very early stage within Network A it was quite clear that the Network needed a set of technical standards by which to judge the adequacy of education outcome indicators. It was felt that such a set of standards were needed because most outcomes data available to the Network were based on surveys that have estimable sampling error and outcome data usually involve the administration of tests, which also involve estimable measurement of error.

Because of these two sources of error it was felt that users of the international outcomes indicators should have some sense of the technical quality of the indicators. Network A has drafted standards for international indicator data, for the purpose of directing a reader to the fact that a standard is not being met and therefore to results that ought to be interpreted with caution. Network A recommends that future international assessment studies be conducted with these standards in mind, and that other indicator projects consider technical standards as part of their reporting practices.

In addition, Network A has significantly expanded its scope to explore ways in which indicators of non curriculum bound outcomes might be developed. In the first instance, this developmental work concentrated on identifying suitable instrumentation and data sources for measuring socio-cultural knowledge and skills, such as basic knowledge required for orientation in the political, social and economic world; problem solving capacity in everyday and critical situations; self perception in the social context; and perception of critical human values. A small subgroup is developing a more detailed plan to identify questions and issues and design a questionnaire on national practices in a small number of countries.

A second area of new work concerns outcomes that derive from the goals for education systems rather than for individuals. This work derives from the perspective that what is an output from one part of the system can be an input for another and vice versa. One of the important outcomes of education systems is what opportunities are provided or offered to students. A second subgroup is

investigating ways in which these system outputs can be systematically defined and measured. The end product of this work may take the form of qualitative rather than quantitative indicators, and thus an important part of this work will be to investigate new ways of presenting data.

Network A has undertaken several research and development activities that will result in indicators for future editions of Education at a Glance that go beyond the current subjects. These activities include:

• A study of national assessment systems, with a view to determining a) what countries assess and how and b) the extent to which these national assessments could provide comparable information on outcomes, or, if found not to be suitable in their present form, how they might be adapted to meet this purpose (with particular attention being given to issues of anchoring and developing an item bank). The project will describe both traditional examination systems serving certification or selection purposes and assessment programs designed to monitor, evaluate, or hold accountable systems or individuals, with an emphasis on the latter. This work is being carried out by Thomas Kellaghan in Ireland.

• The development of indicators for cross-curriculum competencies. The network plans to develop indicators for civic competencies and, finally additional cross-cutting student outcomes. This work is being carried out by the CCC subgroup headed by Uri Trier in Switzerland.

• The development of information about national goals, stated and realized curriculum, and student performance. The initial efforts will include completion of a survey of national goals and a review of stated curriculum. This work is being carried by the GOALS subgroup headed by Marit Granheim in Norway and Sten Petterson in Sweden.

• The preparation of new analyses built around subject-bound data. In particular, Network A is considering gender-based indicators, percent competent and age 9-14 progression in reading literacy, mathematics, and sciences for the third edition of Education at a Glance.

These efforts were in combination with work on subject-bound indicators and standards development for major elements of an emerging strategic plan for Network A activities.

Network A remains cognizant of the limited nature of currently existing cross-national surveys of education outcomes, and supports efforts for OECD to take a more active role in defining its data needs. Network A is also pursuing the development of indicators of non-curriculum bound outcomes. A subgroup, under the leadership of Uri Trier of Switzerland has been charged with the developmental work in this area. The group is considering developing indicators in five areas: knowledge of the social and political world, problem-solving, communication, self perception, and knowledge of democracy. The subgroup has undertaken a preliminary survey to establish the importance of non-curriculum bound outcomes areas in national goals. It has also undertaken a smallscale research project to identify these areas in the curriculum in four countries and commissioned expert papers in each of the five areas of interest.

The second set of education indicators (appearing in the second edition of Education at a Glance, 1993) have come at a time when OECD Member countries face serious problems of sluggish growth and rising unemployment. Because of this situation, in 1992 the OECD Council of Ministers requested that the Secretariat undertake a major study of the causes for, and possible remedies to,

current high levels of unemployment. The study's preliminary findings, presented in Employment/Unemployment Study: Interim Report by the Secretary-General (1993), include human resource development in the range of strategies called for to boost employment and reduce unemployment. The report notes in particular the need for sound initial education, better integration between academic and vocational studies, appropriate linkages and partnerships between schools and employers, workforce relevance of tertiary education, and an adult training system adapted to the needs of employers, workers, and non-workers. The report further recognizes that the relationships between education, training, employment, and productivity growth are complex. The formulation of education policy depends on broad social, economic, and cultural factors.

The monitoring of progress and experimentation in systems of education depends heavily on indicators that enable government authorities and other groups to judge the context and functioning of education and the results achieved. Education indicators can reveal some of the most critical weaknesses of education systems, and can aid the design of corrective policy. Problem areas that need to be addressed are the unequal distribution of opportunities, rigidity of student streaming and tracking systems, and poor use of human resources caused by ineffective education and training programs. The high costs of education are accompanied by discrepancies in per student expenditures. Data presented in Education at a Glance (1993) can contribute to identifying some critical problems and establishing new priorities.

The 1993 edition of Education at a Glance includes economic, social, and demographic context of education; results of education; and information on costs, resources, and processes. Thirty eight indicators offer a body of information on crucial aspects in education policy -- investment levels, financing and staffing, decision-making, level of participation, student tracking, student achievement in key subjects and graduation rates, and whether some levels and types of education give better protection than others against the risks of unemployment.

For each topic, the indicators presented in this publication reveal many similarities and dissimilarities across the countries, and raise questions concerning goals and efficiency of countries' varied educational policies. The indicators also provide important information about new developments in OECD countries' education systems. The primary result of the set of indicators in this edition of Education at a Glance affirms that education and training are part of the problem of a high level of unemployment as well as part of the solution.

Education at a Glance is not a historical study but an instrument for monitoring and guiding educational policy. For this reason the data have to be processed quickly and published without undue delay. Education at a Glance should also reflect reality, which means that the changing conditions of education in a complex international environment must be taken into account. For this reason all the OECD Member countries have engaged in the demanding exercise of updating information, which has involved substantial improvements in the data collection, processing, analysis, and reporting procedures. The year 1993 was devoted to the task of improving the procedures for data collection and transfer. The experience the INES participants have gained in producing the second edition of Education at a Glance is an important resource for achieving a formal and standardized protocol for the regular production of international education indicators. This is one of the major long-term objectives of the INES Project.

The development of the OECD set of international education indicators and the publication of Education at a Glance are generating a series of studies at the national level which complement the international perspective. National indicators studies have been produced or are being developed in Canada, Belgium, Denmark, France, and the United States. This is a very important development

because the policy-relevance and the adequacy of any interpretation of the OECD indicators depends on whether the international data are supplemented by country-specific, within-system information.

The OECD secretariat has received the following reports: The State of the School: 30 indicators of the educational system (Ministry of National Education and Culture, France); Geography of the School (Ministry of National Education and Culture, France); Facts and Figures, Education Indicators Denmark (Ministry of Education and Research, Denmark); The National Education Goals Report, 1992: Building a Nation of Learners (National Education Goals Panel, U.S.A.); How in the World Do Students Read? Effective Schools in Reading: Implications for Educational Planning; and Teaching Reading Around the World (International Association for the Evaluation of Educational Achievement, The Netherlands).

The Policy Review and Advisory Group were joined in their January 1993 meeting by three invited consultants: Erik Hanushek (United States), Claus Moser (United Kingdom), and Francois Orivel (France). These consultants concluded that the first edition of Education at a Glance marks a major step in the development of international education statistics and indicators. They believe that the INES study serves an important function in driving improvement in data collection at the national level. They also identified the need for more adequately addressing the constituencies of industry, business, and the general public; more fully developing indicators concerning outcomes of education; and gathering more information about the contexts -- demographic, cultural and ecological -- in which education systems operate. In addition, they concluded that much value added would be achieved if time series and forecasts were included in a future edition, and that an indicator report must strike a fine balance between complexity and simplicity. They noted that some indicators may be too close to elementary statistics, whereas many others are too complex in the form in which they are shown.

It may become necessary to publish two versions in the future: an indicator report addressed to specialists, and another, much shorter and simpler report intended for the general public. They also stated that clarification is needed on how indicators can be fruitfully employed not merely for informing debate but also for decision-making.

The future of the indicators project depends on the capacity of the OECD to deal with the issue of data collection. In developing new indicators for the third edition of Education at a Glance, the Networks and the Technical Group required data not currently available on a regular basis. Another problem is that some of the data are being provided by different sources. The Policy Review and Advisory Group agreed that it was urgent to address the information needs of the OECD education statistics and indicators activity, and decided to explore ways for the OECD to play an active role in the gathering of education data.

A January 21, 1993 letter from T.J. Alexander, director of the OECD Directorate for Education, Employment, Labour and Social Affairs, to the Education Committee of the CERI Governing Board informed the Members of the outcome of consultations with international organizations to improve the collection, analysis, and publication of CERI indicators and education statistics. OECD, UNESCO, and EUROSTAT have agreed on the need to adapt the ISCED classification system and to revise the common questionnaires in the light of new data requirements that have arisen. With the support of OECD and EUROSTAT, UNESCO will carry principal responsibility for the adaptation of the ISCED system to current needs. In consultation with EUROSTAT and UNESCO, OECD will take the lead in updating and revising the common questionnaires. OECD and EUROSTAT will jointly develop the communication technology needed for the electronic transmission of data from the capitals to the international organizations.

OECD will give priority to the improvement of the indicator templates, and OECD and EUROSTAT have agreed to publish jointly the INES Technical Handbook in late 1994 or early 1995. This will guarantee that the definitions, calculation procedures, and technical criteria used by the organizations are fully compatible. EUROSTAT will carry out a feasibility study concerning the production of education indicators that are particularly suited to the needs of the European Community.

Education at a Glance (first edition), 1992 (30 indicators). Education at a Glance (second edition), 1993. The INES Project will result in successive editions of Education at a Glance (containing results of indicator calculations); a handbook of international educational indicators (containing background information on the development of the project and templates for each indicator); and a set of theoretical and technical papers on various indicator topics.

Making Education Count: Developing and Using International Indicators (summer 1993). A reader bringing together 18 chapters on the development and interpretation of the international education indicators.

INES Technical Handbook (to be published jointly by OECD and EUROSTAT late 1994 or early 1995).

Several compilations of methodological contributions achieved in INES Project Phase 3 (to be published in 1995).

By the time the INES Project is completed in 1996, the Organisation for Economic Co-operation and Development and its Member countries should have succeeded in setting up an international database containing up-to-date and relevant education statistics and indicators; a fully computerized on-line network for collecting and disseminating the data; and cost-effective procedures for processing the data.

Funding Each member country has contributed resources to this project; some have provided substantial additional assistance through their support to the Technical Group, the four Networks, and several ad hoc investigative teams. The publication of Education at a Glance has been facilitated by a special grant to INES by the National Center for Education Statistics in the U.S. Department of Education. Each member country generally carries its own cost of participating in the INES Project. The lead country for each network generally has responsibility for costs incurred in meetings, communication between members, staff support, and publications. The U.S. portion is funded by the National Center for Education Statistics.

International Coordinating Center:

Centre for Educational Research and Innovation

Organisation for Economic Co-operation and Development

2 rue Andre-Pascal

75775 Paris Cedex 16 FRANCE

International Coordinator:

Dr. Norberto Bottani, Principal Administrator INES Project

Centre for Educational Research and Innovation-OECD

2 rue Andre-Pascal

75775 Paris Cedex 16 FRANCE

telephone: 33-1-45-24-92-50

facsimile: 33-1-45-24-90-98

Chair, Consultative Group (Phase 2):

Dr. Herbert Walberg

University of Illinois

522 N. Euclid Avenue

Oak Park, Illinois 60302

telephone: 312/ 996-5580

facsimile: 312/ 996-6400

National Coordinating Center, Network A:

National Center for Education Statistics

555 New Jersey Avenue

Washington, D.C. 20208

National Coordinator:

Dr. Dawn Nelson

International Studies, Data Development Division

National Center for Education Statistics

555 New Jersey Avenue, N.W.

Washington, D.C. 20208

telephone: 202/ 219-1740

facsimile: 202/ 219-1575

Chair, Network A:

Dr. Eugene Owen, Chief, International Activities

Education Assessment Division

555 New Jersey Avenue, N.W.

Washington, D.C. 20208

telephone: 202/ 219-1746

facsimile: 202/ 219-1736

e-mail: Eugene_Owen@doed.gov

Binkley, Marilyn R.

1991 Results of the First International Survey of International and Intranational Educational Outcome Assessment Practices. Paper presented at American Educational Research Association annual meeting, Chicago. April. National Center for Education Statistics, Washington, D.C.

Organisation for Economic Co-Operation and Development/Centre for Educational Research and Innovation

1989 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 2; No. 3 (December). CERI and Education and Training Division of OECD, Paris, France.

1989 OECD/CERI International Indicators Project - Network 2: Education Outcomes Network Report. August.

1989 Project on International Educational Indicators (INES Project): Governing Board Review of Preliminary Results and Proposal for Future Work. Draft. October 7. Paris, France.

1990 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 4 (March). CERI and Education and Training Division of OECD, Paris, France.

1991 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 6 (January); No. 7 (March). CERI and Education and Training Division of OECD, Paris, France.

1991 Outcomes of Education. OECD International Education Indicators. Network A: Report of Phase 2. September.

1992 Education at a Glance: OECD Indicators. OECD, Paris, France.

1992 INES-NEWS: International Indicators and Evaluation of Educational Systems. No. 9 (January); No. 10 (May); No. 12 (April). CERI and Education and Training Division of OECD, Paris, France.

1993 Education at a Glance: OECD Indicators. OECD, Paris, France.

Phillips, Gary W.

1990 OECD/CERI/INES Project - International Education Indicators - Network A: Student Outcomes - Phase Two Working Meeting. Bethesda, Maryland. June 25-27. National Center for Education Statistics, Washington, D.C.

1991 OECD Indicators Project: Network A - Student Outcomes. Meeting Summary, Proposed Indicators, and AERA Presentation on Survey Results. Compiled for the Board on International Comparative Studies in Education. April 22. National Center for Education Statistics, Washington, D.C.

1992 OECD International Education Indicators Project. Status report prepared for the Board on International Comparative Studies in Education. September 11. NCES, Washington, D.C.

1993 OECD INES Project Network A. Status reports prepared for the Board on International Comparative Studies in Education. January 12; May 26. NCES, Washington, D.C.

1993 OECD INES Project. Status report prepared for the Board on International Comparative Studies in Education with Jay Moskowitz and Laura Salganik. October 19. NCES, Washington, D.C.

1994 OECD INES Project. Status report prepared for the Board on International Comparative Studies in Education. January 11. NCES, Washington, D.C.

1993 Technical Standards for International Indicator Data, OECD/CERI International Education Indicator Project. Paper presented at American Educational Research Association annual meeting, Atlanta Georgia. Revised May 25 for presentation to Board on International Comparative Studies in Education June 18. NCES, Washington, D.C.

U.S. Department of Education

1990 OECD/CERI/INES Project, International Education Indicators, Network A: Student Outcomes. Briefing book for meeting. June.

******

NOTE: This study summary was prepared on June 1, 1994 and sent to Gary Phillips (former chair of Network A) and to Eugene Owen (chair of Network A) at the National Center for Education Statistics for review. No suggestions for changes have been made.